Resolution Image Sequences Research Articles

Mid-frequency MTF compensation for optical synthetic aperture based on baseline transform scanning via deep learning

Optical synthetic aperture (OSA) structure of multiple in-phase circular sub-mirrors greatly improves the spatial resolution of space telescopes, but the sparsity and discretization lead to a degradation of mid-frequency response of modulation transfer function (MTF) and loss of image information. A method for improving mid-frequency MTF is presented, named deep learning baseline transform scanning (DLBTS). By performing system baseline transformations, a series of low-resolution imaging sequences containing the missing mid-frequency modulation transfer functions are obtained. Leveraging the powerful feature extraction capability of deep learning, different resolution image sequences are fused to compensate for the missing mid-frequency information. Reverse Regression Module (RRM) is designed to guarantee the loss optimality. When SNR = 30 dB, the PSNR of the Golay-3 image with single-system compensation can be improved from 22.84 dB to 26.29 dB, SSIM can be from 0.685 to 0.765, and MS-SSIM can be raised from 0.865 to 0.916.

Filament displacement image analytics tool for use in investigating dynamics of dense microtubule networks.

The fate and motion of cells is influenced by a variety of physical characteristics of their microenvironments. Traditionally, mechanobiology focuses on external mechanical phenomena such as cell movement and environmental sensing. However, cells are inherently dynamic, where internal waves and internal oscillations are a hallmark of living cells observed under a microscope. We propose that these internal mechanical rhythms provide valuable information about cell health. Therefore, it is valuable to capture the rhythms inside cells and quantify how drugs or physical interventions affect a cell's internal dynamics. One of the key dynamical entities inside cells is the microtubule network. Typically, microtubule dynamics are measured by end-protein tracking. In contrast, this paper introduces an easy-to-implement approach to measure the lateral motion of the microtubule filaments embedded within dense networks with (at least) confocal resolution image sequences. Our tool couples the computer vision algorithm Optical Flow with an anisotropic, rotating Laplacian of Gaussian filtering to characterize the lateral motion of dense microtubule networks. We then showcase additional image analytics used to understand the effect of microtubule orientation and regional location on lateral motion. We argue that our tool and these additional metrics provide a fuller picture of the active forcing environment within cells.

Video-Based Eye Blink Identification and Classification.

Blink detection and classification can provide a very useful clinical indicator, because of its relation with many neurological and ophthalmological conditions. In this work, we propose a system that automatically detects and classifies blinks as "complete" or "incomplete" in high resolution image sequences zoomed into the participants' face, acquired during clinical examination using near-Infrared illumination. This method utilizes state-of-the-art (DeepLabv3+) deep learning encoder-decoder neural architecture -DLED to segment iris and eyelid in both eyes in the acquired images. The sequence of the segmented frames is post-processed to calculate the distance between the eyelids of each eye (palpebral fissure height) and the corresponding iris diameter. These quantities are temporally filtered and their fraction is subject to adaptive thresholding to identify blinks and determine their type, independently for each eye. The proposed system was tested on 15 participants, each with one video of 4 to 10 minutes. Several metrics of blink detection and classification accuracy were calculated against the ground truth, which was generated by three (3) independent experts, whose conflicts were resolved by a senior expert. Results show that the proposed system achieved F1-score 95.3% and 80.9% for the classification of complete and incomplete blinks respectively, collectively for all 15 participants, outperforming all 3 experts. The proposed system was proven robust in handling unexpected participant movements and actions, as well as glare and reflections from the spectacles, or face obstruction by facemasks.

Estimation of High Framerate Digital Subtraction Angiography Sequences at Low Radiation Dose.

Digital Subtraction Angiography (DSA) provides high resolution image sequences of blood flow through arteries and veins and is considered the gold standard for visualizing cerebrovascular anatomy for neurovascular interventions. However, acquisition frame rates are typically limited to 1-3 fps to reduce radiation exposure, and thus DSA sequences often suffer from stroboscopic effects. We present the first approach that permits generating high frame rate DSA sequences from low frame rate acquisitions eliminating these artifacts without increasing the patient's exposure to radiation. Our approach synthesizes new intermediate frames using a phase-aware Convolutional Neural Network. This network accounts for the non-linear blood flow progression due to vessel geometry and initial velocity of the contrast agent. Our approach out-performs existing methods and was tested on several low frame rate DSA sequences of the human brain resulting in sequences of up to 17 fps with smooth and continuous contrast flow, free of flickering artifacts.

Spatio-Temporal Segmentation Applied to Optical Remote Sensing Image Time Series

The availability of a large amount of remote sensing data made Earth Observation increasingly accessible and detailed. High temporal and spatial resolution sensors are responsible for making available data sets of time series in unprecedented proportions. Within this context, the use of efficient segmentation algorithms of remote sensing imagery represents an important role in this scenario, because they provide homogeneous regions in space-time and hence simplify the data set. In addition, the spatio-temporal segmentation can bring a new way of interpreting data by means of analyzing contiguous regions in time. This letter describes a method for image segmentation applied to time series of the Earth Observation data. We adapted the traditional region growing method to detect homogeneous regions in space and time. Study cases were conducted by considering the dynamic time warping algorithm as the homogeneity criterion to grow regions. Tests on high temporal resolution image sequences from Moderate Resolution Imaging Spectroradiometer and Landsat-8 Operational Land Imager vegetation indices and comparisons with other distance measurements provided satisfactory outcomes.

Simultaneous Temporal Superresolution and Denoising for Cardiac Fluorescence Microscopy

Due to low-light emission of fluorescent samples, live fluorescence microscopy imposes a tradeoff between spatio-temporal resolution and signal-to-noise ratio. This can result in images and videos containing motion blur or Poisson-type shot noise, depending on the settings used during acquisition. Here, we propose an algorithm to simultaneously denoise and temporally superresolve movies of repeating microscopic processes that is compatible with any conventional microscopy setup that can achieve imaging at a rate of at least twice that of the fundamental frequency of the process (above 4 frames per second for a 2-Hz process). Our method combines low temporal resolution frames from multiple cycles of a repeating process to reconstruct a denoised, higher temporal resolution image sequence which is the solution to a linear program that maximizes the consistency of the reconstruction with the measurements, under a regularization constraint. This paper describes, in particular, a parallelizable superresolution reconstruction algorithm and demonstrates its application to live cardiac fluorescence microscopy. Using our method, we experimentally show temporal resolution improvement by a factor of 1.6, resulting in a visible reduction of motion blur in both on-sample and off-sample frames.

Frame rate required for speckle tracking echocardiography: A quantitative clinical study with open-source, vendor-independent software

ObjectivesTo determine the optimal frame rate at which reliable heart walls velocities can be assessed by speckle tracking. BackgroundAssessing left ventricular function with speckle tracking is useful in patient diagnosis but requires a temporal resolution that can follow myocardial motion. In this study we investigated the effect of different frame rates on the accuracy of speckle tracking results, highlighting the temporal resolution where reliable results can be obtained. Material and methods27 patients were scanned at two different frame rates at their resting heart rate. From all acquired loops, lower temporal resolution image sequences were generated by dropping frames, decreasing the frame rate by up to 10-fold. ResultsTissue velocities were estimated by automated speckle tracking. Above 40 frames/s the peak velocity was reliably measured. When frame rate was lower, the inter-frame interval containing the instant of highest velocity also contained lower velocities, and therefore the average velocity in that interval was an underestimate of the clinically desired instantaneous maximum velocity. ConclusionsThe higher the frame rate, the more accurately maximum velocities are identified by speckle tracking, until the frame rate drops below 40 frames/s, beyond which there is little increase in peak velocity. We provide in an online supplement the vendor-independent software we used for automatic speckle-tracked velocity assessment to help others working in this field.

Characterizing motion types of G-band bright points in the quiet Sun**Supported by the National Natural Science Foundation of China.

We study the motion of G-band bright points (GBPs) in the quiet Sun to obtain the characteristics of different motion types. A high resolution image sequence taken with the Hinode/Solar Optical Telescope (SOT) is used, and GBPs are automatically tracked by segmenting 3D evolutional structures in a space-time cube. After putting the GBPs that do not move during their lifetimes aside, the non-stationary GBPs are categorized into three types based on an index of their motion type. Most GBPs that move in straight or nearly straight lines are categorized as a straight motion type, a few moving in rotary paths into rotary motion, and the others fall into a motion type we called erratic. The mean horizontal velocities are 2.18±0.08 km s−1, 1.63±0.09 km s−1 and 1.33±0.07 km s−1 for straight, erratic and rotary motion types, respectively. We find that a GBP drifts at a higher and constant velocity during its whole life if it moves in a straight line. However, it has a lower and variational velocity if it moves on a rotary path. The diffusive process is ballistic-, super- and sub-diffusion for straight, erratic and rotary motion types, respectively. The corresponding diffusion index (γ) and coefficients (K) are 2.13±0.09 and 850±37 km2 s−1, 1.82±0.07 and 331±24 km2 s−1, and 0.73±0.19 and 13±9 km2 s−1. In terms of direction of motion, it is homogeneous and isotropic, and usually persists between neighboring frames, no matter what motion type a GBP is classified as.

Evolution of isolated G-band bright points: size, intensity and velocity

We study the evolution pattern of isolated G-band bright points (GBPs) in terms of their size, intensity and velocity. Using a high resolution image sequence taken with the Hinode/Solar Optical Telescope (SOT), we detect GBPs in each image by the Laplacian and Morphological Dilation algorithm, and track their evolutions by a 26-adjacent method in a three-dimensional space-time cube. For quantifying the evolution, we propose a quantification method based on lifetime normalization which aligns the different lifetimes to common stages. The quantification results show that, on average, the diameter of isolated GBPs changes from 173 to 166 km, then down to 165 km; the maximum intensity contrast changes from 1.012 to 1.027, then down to 1.011; however, the velocity changes from 1.709 to 1.593 km s−1, then up to 1.703 km s−1. The results indicate that the evolution follows a pattern such that the GBP is small, faint and fast-moving at the birth stage, becomes big, bright and slow-moving at the middle stage, then gets small, faint and fast-moving at the decay stage until disappearance. Although the differences are very small, a two-sample t-test is used to demonstrate there are significant differences in means between the distributions of the different stages. Furthermore, we quantify the relationship between the lifetimes of GBPs and their properties. It is found that there are positive correlations between the lifetimes and their sizes and intensities with correlation coefficients of 0.83 and 0.65, respectively; however, there is a negative correlation between the lifetimes and velocities with a correlation coefficient of −0.49. In summary, the longer the GBP persists, the bigger, brighter and slower it will be.

Detection and Tracking of Vehicle Target Based on Super-Resolution Reconstruction and Variable Template Matching

Vehicle Target Detection and Tracking Method Based on Image Super-Resolution Reconstruction and Variable Template Matching is Put Forward. Firstly, a Nonlinear Iterative Algorithm is Applied to Reconstruct a Super-Resolution Image from Low Resolution Image Sequence; then, the Image is Standardized and the Movement Areas are Determined; Finally, the Variable Template Matching Method is Used to Detect and Track the Vehicle Targets in Movement Areas. from the Characteristics of Algorithm and the Experiment Results, we can see that the Proposed Algorithm Improves the Matching Accuracy of Target Tracking and Better Solves the Limitation of Missed Detection for Traditional Methods. the Reason of the Good Performance of the Proposed Algorithm Relies in High Quality Images Acquired by Super-Resolution Reconstruction from Low Resolution Image Sequence and the Application of Variable Template Matching Method.

Migration of Ca II H bright points in the internetwork

The migration of magnetic bright point-like features (MBP) in the lower solar atmosphere reflects the dispersal of magnetic flux as well as the horizontal flows of the atmospheric layer they are embedded in. We analyse trajectories of the proper motion of intrinsically magnetic, isolated internetwork Ca II H MBPs (mean lifetime 461 +- 9 s) to obtain their diffusivity behaviour. We use seeing-free high spatial and temporal resolution image sequences of quiet-Sun, disc-centre observations obtained in the Ca II H 3968 {\AA} passband of the Sunrise Filter Imager (SuFI) onboard the Sunrise balloon-borne solar observatory. Small MBPs in the internetwork are automatically tracked. The trajectory of each MBP is then calculated and described by a diffusion index ({\gamma}) and a diffusion coefficient (D). We further explore the distribution of the diffusion indices with the help of a Monte Carlo simulation. We find {\gamma} = 1.69 +- 0.08 and D = 257 +- 32 km^2/s averaged over all MBPs. Trajectories of most MBPs are classified as super-diffusive, i.e., {\gamma} > 1, with the determined {\gamma} being to our knowledge the largest obtained so far. A direct correlation between D and time-scale ({\tau}) determined from trajectories of all MBPs is also obtained. We discuss a simple scenario to explain the diffusivity of the observed, relatively short-lived MBPs while they migrate within a small area in a supergranule (i.e., an internetwork area). We show that the scatter in the {\gamma} values obtained for individual MBPs is due to their limited lifetimes. The super-diffusive MBPs can be well-described as random walkers (due to granular evolution and intergranular turbu- lence) superposed on a large systematic (background) velocity, caused by granular, mesogranular and supergranular flows.

Super-resolution reconstruction face recognition based on multi-level FFD registration

Super-resolution (SR) reconstruction is an effective method to solve the problem, that the face image resolution is too low to be recognized in video, but the non-rigid change of deformed face and expression changes greatly affect the accuracy of registration and reconstruction. To solve these problems, a method of multi-level model free form deformation (FFD) elastic registration algorithm based on B spline is proposed. It first use low-resolution FFD grid for global registration, to emphasize the contribution of edge information for registration, we introduce edge registration measure into the sum of squared difference (SSD) criterion. Then divide the global registration image and reference image into a series of corresponding sub-image pairs and calculate the correlation coefficient of each pair; at the same time, we do local registration with high-resolution FFD grid to the small value correlation coefficient sub-image pairs. In the registration process of optimization, the paper uses adaptive step length gradient descent method algorithm based on chaotic variables to improve optimization efficiency. After registration, the algorithm of project onto convex sets (POCS) is used to reconstruct SR face image through several low resolution image sequences, and then recognized these SR face images by support vector machines (SVM) classifier. Experimental results from standard video database and self-built video database show that this method can register and reconstruct face image accurately in the condition of great face deformation and expression change, while the face recognition accuracy is also improved.

Reactive sintering: An important component in the combustion of nanocomposite thermites

One of the open questions in understanding the reactivity of nanometric metal/metal oxide particulate composites is the relative role of gas–solid vs. condensed state reactions. This work is an investigation of several nano-Al based thermites subjected to very rapid heating rates. The ignition temperature of thermites, as measured by the onset of optical emission, was measured using a rapidly heated Pt wire. Generally, ignition was seen to occur above the melting temperature of aluminum. The exception was Al/Bi 2O 3 which ignited slightly below this point. Samples were also rapidly heated in situ within electron microscopes to provide direct imaging before and after heating. The sintering of aggregated and/or agglomerated particles into characteristically larger structures was experimentally observed in all cases, and the fuel and oxidizer were found to be in surface contact suggesting that a condensed-phase reactive sintering mechanism had occurred. High resolution image sequences of thermites ignited on the Pt wire were collected using a real time phase contrast imaging technique at the Advanced Photon Source of Argonne National Lab. The results suggest that reactive sintering occurs on a fast timescale, and relatively early in the reaction, leading to rapid melting and coalescence of aggregated particles. This dramatically changes the initial size and morphology of the constituents before the remainder of the material burns. The results question the idea that a decrease in particle size will necessarily lead to an enhancement in reactivity, since large amounts of sintering occurs early in the reaction, and alters the morphology. It is suggested that improvements in reactivity can be achieved by designing architectures to improve the interfacial contact upon sintering, as well as by selecting oxidizers based on their ability to liberate and transport oxygen in the condensed phase, while producing volatile species to assist in convective energy transport.

Quasi-periodic polar flares at Jupiter: A signature of pulsed dayside reconnections?

[1] The most dynamic part of the Jovian UV aurora is located inside the main auroral oval. This region is known to regularly show localized but dramatic enhancements on timescales of several tens of seconds, called polar flares. They have often been associated with the polar cusp, based on their location in the polar cap. The present study is based on the longest high-time resolution image sequences ever acquired by the Space Telescope Imaging Spectrograph aboard the Hubble Space Telescope. We report the first observations of a regularity in the occurrence of these flares, with a timescale of 2–3 minutes. We use a magnetic flux mapping model to identify the region corresponding to these emissions in the equatorial plane: the radial distance ranges from 55 to 120 Jovian radii and the local times are between 10:00 and 18:00. The analogy with similar phenomena observed at Earth suggests that these quasi-periodic auroral flares could be related to pulsed reconnections at the dayside magnetopause. Indeed, the flares' projected location in the equatorial plane and their rate of re-occurrence show some similarities with the properties of the flux transfer events observed by the Pioneer and Voyager probes.

Multiple Distortion Measures for Packetized Scalable Media

As the diversity in end-user devices and networks grows, it becomes important to be able to efficiently and adaptively serve media content to different types of users. A key question surrounding adaptive media is how to do Rate-Distortion optimized scheduling. Typically, distortion is measured with a single distortion measure, such as the Mean-Squared Error compared to the original high resolution image or video sequence. Due to the growing diversity of users with varying capabilities such as different display sizes and resolutions, we introduce <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Multiple</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Distortion</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Measures</i> (MDM) to account for a diverse range of users and target devices. MDM gives a clear framework with which to evaluate the performance of media systems which serve a variety of users. Scalable coders, such as JPEG2000 and H.264/MPEG-4 SVC, allow for adaptation to be performed with relatively low computational cost. We show that accounting for MDM can significantly improve system performance; furthermore, by combining this with scalable coding, this can be done efficiently. Given these MDM, we propose an algorithm to generate <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">embedded</i> schedules, which enables low-complexity, adaptive streaming of scalable media packets to minimize distortion across multiple users. We show that using MDM achieves up to 4 dB gains for spatial scalability applied to images and 12 dB gains for temporal scalability applied to video.

A study of spicules from space observations

AbstractWe have studied spicules observed at the northern solar limb by using simultaneous high resolution image sequences. The images were obtained by Hinode/SOT (in the Ca II H passband) and TRACE (in the 1600 Å passband) during a coordinated campaign. Both data sets were reduced and then carefully co-aligned in order to compare the observed patterns in this highly dynamic region of the Sun. The identification of individual structures in both spectral bands allows us to trace their spatial and temporal behaviour. Persistent intensity variations at certain locations, indicate that at least some spicules have a recurrent behavior. Using wavelet analysis we investigate oscillatory phenomena along the axis of off-limb spicules and we construct 2-D maps of the solar limb with the observed oscillations.

People detection in low-resolution video with non-stationary background

In this paper, we present a framework for robust people detection in low resolution image sequences of highly cluttered dynamic scenes with non-stationary background. Our model utilizes appearance features together with short- and long-term motion information. In particular, we boost Integral Gradient Orientation histograms of appearance and short-term motion. Outputs from the detector are maintained by a tracker to correct any misdetections. A Bayesian model is then deployed to further fuse long-term motion information based on correlation. Experiments show that our model is more robust with better detection rate compared to the model of Viola et al. [Michael J. Jones Paul Viola, Daniel Snow, Detecting pedestrians using patterns of motion and appearance, International Journal of Computer Vision 63(2) (2005) 153–161].

Region-based super-resolution for compression

Every user of multimedia technology expects good image and video visual quality independently of the particular characteristics of the receiver or the communication networks employed. Unfortunately, due to factors like processing power limitations and channel capabilities, images or video sequences are often downsampled and/or transmitted or stored at low bitrates, resulting in a degradation of their final visual quality. In this paper, we propose a region-based framework for intentionally introducing downsampling of the high resolution (HR) image sequences before compression and then utilizing super resolution (SR) techniques for generating an HR video sequence at the decoder. Segmentation is performed at the encoder on groups of images to classify their blocks into three different types according to their motion and texture. The obtained segmentation is used to define the downsampling process at the encoder and it is encoded and provided to the decoder as side information in order to guide the SR process. All the components of the proposed framework are analyzed in detail. A particular implementation of it is described and tested experimentally. The experimental results validate the usefulness of the proposed method.