Consecutive Images Research Articles

Aberrant Developmental Patterns of Gamma-Band Response and Long-Range Communication Disruption in Youths With 22q11.2 Deletion Syndrome.

Brain oscillations play a pivotal role in synchronizing responses of local and global ensembles of neurons. Patients with schizophrenia exhibit impairments in oscillatory response, which are thought to stem from abnormal maturation during critical developmental stages. Studying individuals at genetic risk for psychosis, such as 22q11.2 deletion carriers, from childhood to adulthood may provide insights into developmental abnormalities. The authors acquired 106 consecutive T1-weighted MR images and 40-Hz auditory steady-state responses (ASSRs) with high-density (256 channel) EEG in a group of 58 22q11.2 deletion carriers and 48 healthy control subjects. ASSRs were analyzed with 1) time-frequency analysis using Morlet wavelet decomposition, 2) intertrial phase coherence (ITPC), and 3) theta-gamma phase-amplitude coupling estimated in the source space between brain regions activated by the ASSRs. Additionally, volumetric analyses were performed with FreeSurfer. Subanalyses were conducted in deletion carriers who endorsed psychotic symptoms and in subgroups with different age bins. Deletion carriers had decreased theta and late-latency 40-Hz ASSRs and phase synchronization compared with control subjects. Deletion carriers with psychotic symptoms displayed a further reduction of gamma-band response, decreased ITPC, and decreased top-down modulation of gamma-band response in the auditory cortex. Reduced gamma-band response was correlated with the atrophy of auditory cortex in individuals with psychotic symptoms. In addition, a linear increase of theta and gamma power from childhood to adulthood was found in control subjects but not in deletion carriers. The results suggest that while all deletion carriers exhibit decreased gamma-band response, more severe local and long-range communication abnormalities are associated with the emergence of psychotic symptoms and gray matter loss. Additionally, the lack of age-related changes in deletion carriers indexes a potential developmental impairment in circuits underlying the maturation of neural oscillations during adolescence. The progressive disruption of gamma-band response in 22q11.2 deletion syndrome supports a developmental perspective toward understanding and treating psychotic disorders.

Solar Wind Velocities at Comets C/2011 L4 Pan‐STARRS and C/2013 R1 Lovejoy Derived Using a New Image Analysis Technique

AbstractThe ion tails of bright comets have long been considered as a natural tracers of the solar wind (SW) near these objects. Studies of comets and their ion tails allow inexpensive monitoring of key SW structures in the inner heliosphere, much of which is otherwise only accessible by in situ SW spacecraft measurements. Here, we present a novel technique to mine the rich archive of amateur, professional and spacecraft observations of cometary ion tails. To demonstrate this, we focus on Near‐Sun comet C/2011 L4 (Pan‐STARRS) during Carrington Rotations (CR) 2134 and 2135 and comet C/2013 R1 (Lovejoy) during CR 2118. We outline the technique’s shortcomings, including its geometric limitations, and present a catalog of radial SW velocities derived in the near‐comet environment and information on the heliospheric conditions inferred from the measured SW. Complementary measurements, derived from folding ion rays and a velocity profile map built from consecutive images, are provided as an alternative means of quantifying the SW ‐cometary ionosphere interaction. We find that comets are generally good indicators of SW structure, but the quality of the results is strongly dependent on the observing geometry.

Video Super Resolution Using a Selective Edge Aggregation Network

An edge map is a feature map representing the contours of the object in the image. There was a Single Image Super Resolution (SISR) method using the edge map, which achieved a notable SSIM performance improvement. Unlike SISR, Video Super Resolution (VSR) uses video, which consists of consecutive images with temporal features. Therefore, some VSR models adopted motion estimation and motion compensation to apply spatio-temporal feature maps. Unlike the models above, we tried a different method by adding edge structure information and its related post-processing to the existing model. Our model “Video Super Resolution Using a Selective Edge Aggregation Network (SEAN)” consists of a total of two stages. First, the model selectively generates an edge map using the target frame and also the neighboring frame. At this stage, we adopt the magnitude loss function so that the output of SEAN more clearly learns the contours of each object. Second, the final output is generated using the refinement (post-processing) module. SEAN shows more distinct object contours and better color correction compared to other existing models.

The Thresher: Lucky imaging without the waste

ABSTRACT In traditional lucky imaging (TLI), many consecutive images of the same scene are taken with a high frame-rate camera, and all but the sharpest images are discarded before constructing the final shift-and-add image. Here, we present an alternative image analysis pipeline – The Thresher – for these kinds of data, based on online multi-frame blind deconvolution. It makes use of all available data to obtain the best estimate of the astronomical scene in the context of reasonable computational limits; it does not require prior estimates of the point-spread functions in the images, or knowledge of point sources in the scene that could provide such estimates. Most importantly, the scene it aims to return is the optimum of a justified scalar objective based on the likelihood function. Because it uses the full set of images in the stack, The Thresher outperforms TLI in signal-to-noise ratio; as it accounts for the individual-frame PSFs, it does this without loss of angular resolution. We demonstrate the effectiveness of our algorithm on both simulated data and real Electron-Multiplying CCD images obtained at the Danish 1.54-m telescope (hosted by ESO, La Silla). We also explore the current limitations of the algorithm, and find that for the choice of image model presented here, non-linearities in flux are introduced into the returned scene. Ongoing development of the software can be viewed at https://github.com/jah1994/TheThresher.

A Limited-View CT Reconstruction Framework Based on Hybrid Domains and Spatial Correlation.

Limited-view Computed Tomography (CT) can be used to efficaciously reduce radiation dose in clinical diagnosis, it is also adopted when encountering inevitable mechanical and physical limitation in industrial inspection. Nevertheless, limited-view CT leads to severe artifacts in its imaging, which turns out to be a major issue in the low dose protocol. Thus, how to exploit the limited prior information to obtain high-quality CT images becomes a crucial issue. We notice that almost all existing methods solely focus on a single CT image while neglecting the solid fact that, the scanned objects are always highly spatially correlated. Consequently, there lies bountiful spatial information between these acquired consecutive CT images, which is still largely left to be exploited. In this paper, we propose a novel hybrid-domain structure composed of fully convolutional networks that groundbreakingly explores the three-dimensional neighborhood and works in a “coarse-to-fine” manner. We first conduct data completion in the Radon domain, and transform the obtained full-view Radon data into images through FBP. Subsequently, we employ the spatial correlation between continuous CT images to productively restore them and then refine the image texture to finally receive the ideal high-quality CT images, achieving PSNR of 40.209 and SSIM of 0.943. Besides, unlike other current limited-view CT reconstruction methods, we adopt FBP (and implement it on GPUs) instead of SART-TV to significantly accelerate the overall procedure and realize it in an end-to-end manner.

Application of consecutive polyethylene glycol treatments for modeling the seminal root growth of rice under water stress

The response of root growth to various osmotic potentials is quite important in assessing the drought resistance of rice. This study modeled seminal root growth by applying consecutive polyethylene glycol (PEG) treatments (from 0 to 25%, 1% step), mathematical equations and noncontact image analysis to quantitatively evaluate the root response. Treatment began after seeds were germinated, and root growth was recorded by a digital camera every day from 7 to 20 days after seeding (DAS). Although the seminal root length (SRL) measured by image analysis slightly varied with DAS, the equations explained the differences in SRL increases under each PEG concentration relatively well (R2 = 0.774). The equations also suggested that the maximum seminal root length was observed at 5.9% PEG. This numerical characterization of root growth is an effective means of evaluating drought resistance.

Fast and incremental loop closure detection with deep features and proximity graphs

AbstractIn recent years, the robotics community has extensively examined methods concerning the place recognition task within the scope of simultaneous localization and mapping applications. This article proposes an appearance‐based loop closure detection pipeline named “Fast and Incremental Loop closure Detection (FILD++). First, the system is fed by consecutive images and, via passing them twice through a single convolutional neural network, global and local deep features are extracted. Subsequently, a hierarchical navigable small‐world graph incrementally constructs a visual database representing the robot's traversed path based on the computed global features. Finally, a query image, grabbed each time step, is set to retrieve similar locations on the traversed route. An image‐to‐image pairing follows, which exploits local features to evaluate the spatial information. Thus, in the proposed article, we propose a single network for global and local feature extraction in contrast to our previous work (FILD), while an exhaustive search for the verification process is adopted over the generated deep local features avoiding the utilization of hash codes. Exhaustive experiments on eleven publicly available data sets exhibit the system's high performance (achieving the highest recall score on eight of them) and low execution times (22.05 ms on average in New College, which is the largest one containing 52,480 images) compared to other state‐of‐the‐art approaches.

An Elastic Frame Rate Up-Conversion for Sequential Omnidirectional Images

A distinct innovation in the development of frame rate up-conversion for an omnidirectional video is presented in this paper. The innovation allows an omnidirectional video to decide the number of omnidirectional frames that the main algorithm can create based on the characteristic of the motion of objects in the omnidirectional video itself. This elastic omnidirectional frame production is achievable since the main algorithm works based on an optical flow method with a self-improvement mechanism. The optical flow concept has proven to be adaptable greatly with the character of such a video. The experiment results, in which ten omnidirectional videos were involved, show that the algorithm's exertion and adaptability named the elastic frame rate up-conversion (E-FRUC) are incredible. This achievement is confirmed by both the excellent visual quality and the high score of the PSNR (33 to 59 dB) of the reconstructed omnidirectional frame (RF). The E-FRUC can generate at least one RF from two consecutive omnidirectional images, although these images are synthetically designed with an ideal motion. The more complex the motion objects condition inside the omnidirectional video is, the more RFs the E-FRUC can create. By using the E-FRUC, the continuity of motion of moving objects can be preserved. Therefore it can be useful to maintain the frame rate of incomplete omnidirectional video frames when such video is played back. Such a condition usually happens when the omnidirectional video is transmitted through error-prone telecommunication networks.

Spatial-temporal 3D dependency matching with self-supervised deep learning for monocular visual sensing

Monocular visual sensing is the task of using a camera to estimate the scene depth, optical flow and camera pose. In this paper, we propose a spatial–temporal 3D dependency matching approach that enforces the robustness of continuous frames matching for monocular visual sensing. 3D structure and warped depth based geometry backpropagation are used to encourage jointly learning the view depth, optical flow and camera pose employing a novel self-supervised neural network from monocular sequences. We designed two different iterative convolutional prediction sub-networks, where the optical flow obtained by depth and camera pose is iteratively used for depth prediction. A virtual frame method is proposed to optimize the optical flow of moving objects. The salient feature of the proposed learning framework is completely unsupervised, requiring only consecutive monocular images for training and testing. Evaluation on publicly benchmark datasets shows that our unsupervised learning model significantly outperforms previous methods and achieves better performance compared with previously unsupervised manners and achieves comparable results with supervised ones.

Improving the Model for Person Detection in Aerial Image Sequences Using the Displacement Vector: A Search and Rescue Scenario

Recent results in person detection using deep learning methods applied to aerial images gathered by Unmanned Aerial Vehicles (UAVs) have demonstrated the applicability of this approach in scenarios such as Search and Rescue (SAR) operations. In this paper, the continuation of our previous research is presented. The main goal is to further improve detection results, especially in terms of reducing the number of false positive detections and consequently increasing the precision value. We present a new approach that, as input to the multimodel neural network architecture, uses sequences of consecutive images instead of only one static image. Since successive images overlap, the same object of interest needs to be detected in more than one image. The correlation between successive images was calculated, and detected regions in one image were translated to other images based on the displacement vector. The assumption is that an object detected in more than one image has a higher probability of being a true positive detection because it is unlikely that the detection model will find the same false positive detections in multiple images. Based on this information, three different algorithms for rejecting detections and adding detections from one image to other images in the sequence are proposed. All of them achieved precision value about 80% which is increased by almost 20% compared to the current state-of-the-art methods.

Predictive role of optical coherence tomography angiography for exudation recurrence in patients with type 1 neovascular age-related macular degeneration treated with pro-re-nata protocol.

We sought to identify the consecutive changes and predictive features for exudation recurrence in macular neovascularization (MNV) using optical coherence tomography angiography (OCTA) in type 1 neovascular age-related macular degeneration (NVAMD). A total of 291 OCTA images in consecutive visit of 45 patients newly diagnosed with type 1 NMV and treated with three loading intravitreal anti-vascular endothelial growth factor injections (IVIs) and a pro-re-nata (PRN) therapy regimen were analysed. Quantitative features of OCTA included the MNV area, MNV length, total number of endpoints (open-ended vessels) and junctions (internal branching) using AngioTool. Two subgroups were divided according to exudation recurrence time from the third IVI (group 1: ≤3 months vs. group 2: >3 months). The area, length, number of total junctions, and endpoints decreased during three loading IVIs and increased at exudation recurrence (all p < 0.05). In a subgroup analysis of consecutive OCTA images, the number of total endpoints increased at two months prior to exudate recurrence in group 2 (the late recurrence group, p = 0.020). A higher total number of endpoints of MNV at baseline were found to be related with group 1 (early recurrence, p = 0.020 and 0.012 in univariate and multivariate regression analyses). The MNV with higher open-ended vessels at the lesion periphery at baseline might be expected to show earlier recurrence of exudation after loading IVIs. By observing the number of open-ended vessels in consecutive OCTA images, exudation recurrence could be predicted.

Artificial intelligence for stepwise diagnosis and monitoring of COVID-19.

BackgroundMain challenges for COVID-19 include the lack of a rapid diagnostic test, a suitable tool to monitor and predict a patient’s clinical course and an efficient way for data sharing among multicenters. We thus developed a novel artificial intelligence system based on deep learning (DL) and federated learning (FL) for the diagnosis, monitoring, and prediction of a patient’s clinical course.MethodsCT imaging derived from 6 different multicenter cohorts were used for stepwise diagnostic algorithm to diagnose COVID-19, with or without clinical data. Patients with more than 3 consecutive CT images were trained for the monitoring algorithm. FL has been applied for decentralized refinement of independently built DL models.ResultsA total of 1,552,988 CT slices from 4804 patients were used. The model can diagnose COVID-19 based on CT alone with the AUC being 0.98 (95% CI 0.97–0.99), and outperforms the radiologist’s assessment. We have also successfully tested the incorporation of the DL diagnostic model with the FL framework. Its auto-segmentation analyses co-related well with those by radiologists and achieved a high Dice’s coefficient of 0.77. It can produce a predictive curve of a patient’s clinical course if serial CT assessments are available.InterpretationThe system has high consistency in diagnosing COVID-19 based on CT, with or without clinical data. Alternatively, it can be implemented on a FL platform, which would potentially encourage the data sharing in the future. It also can produce an objective predictive curve of a patient’s clinical course for visualization.Key Points• CoviDet could diagnose COVID-19 based on chest CT with high consistency; this outperformed the radiologist’s assessment. Its auto-segmentation analyses co-related well with those by radiologists and could potentially monitor and predict a patient’s clinical course if serial CT assessments are available. It can be integrated into the federated learning framework.• CoviDet can be used as an adjunct to aid clinicians with the CT diagnosis of COVID-19 and can potentially be used for disease monitoring; federated learning can potentially open opportunities for global collaboration.Supplementary InformationThe online version contains supplementary material available at 10.1007/s00330-021-08334-6.

Fast-Yolo-Rec: Incorporating Yolo-Base Detection and Recurrent-Base Prediction Networks for Fast Vehicle Detection in Consecutive Images

Fast-Yolo-Rec: Incorporating Yolo-Base Detection and Recurrent-Base Prediction Networks for Fast Vehicle Detection in Consecutive Images

Variational image motion estimation by preconditioned dual optimization

&lt;p style='text-indent:20px;'&gt;Estimating optical flows is one of the most interesting problems in computer vision, which estimates the essential information about pixel-wise displacements between two consecutive images. This work introduces an efficient dual optimization framework with accelerated preconditioners to the challenging nonsmooth optimization problem of total-variation regularized optical-flow estimation. In theory, the proposed dual optimization framework brings an elegant variational analysis to the given difficult optimization problem, while presenting an efficient algorithmic scheme without directly tackling the corresponding nonsmoothness in numeric. By introducing efficient preconditioners with a multi-scale implementation, the proposed preconditioned dual optimization approaches achieve competitive estimation results of image motion, compared to the state-of-the-art methods. Moreover, we show that the proposed preconditioners can guarantee convergence of the implemented numerical schemes with high efficiency.&lt;/p&gt;

Dynamic Monitoring of Mining Destruction on Coal Seam Floor With Constrained Time-Lapse Resistivity Imaging Inversion

With the increase in coal mining depth, water hazards in deep, strong karst aquifers have seriously threatened the safety of coal mine production. Dynamic monitoring of floor damage during coal seam mining under complex conditions is of great significance for the prediction of water inrush at the working face and for improving coal mining methods. Based on parallel electrical monitoring technology, a constrained time-lapse resistivity inversion method was adopted to process the electrical data of the working face floor. The background resistivity inversion results were used as an a priori model to act on the inversion of subsequent monitoring resistivity data, and the subsequent inversion results of resistivity were fed back to the prior model, which greatly eliminated inversion artifacts. Finally, the dynamic change process of the resistivity profile before and after the failure of the bottom floor of the mining face was determined, and the maximum depth of the failure zone of the floor is 25m, which is located above the lower aquifer. The field monitoring data show that compared with the method of directly comparing the differences between consecutive monitoring images, the inversion profiles obtained by the constrained time-lapse resistivity inversion method have clearer differences in the spatial and temporal dimensions. The depth of the water-conducting damage zone of the floor was clarified, and the basic development characteristic of the floor damage zone of the working face was obtained, which has practical application value for ensuring the safe mining of the working face.

InSAR Phase Unwrapping Error Correction for Rapid Repeat Measurements of Water Level Change in Wetlands

Here, we present an enhanced algorithm to correct interferometric synthetic aperture radar (InSAR) phase unwrapping errors by incorporating iterative spatial bridging between islands and phase closure among interferograms. We use rapid repeat airborne synthetic aperture radar acquisitions from NASA’s airborne uninhabited aerial vehicle synthetic aperture radar (UAVSAR) instrument to estimate short-term changes in water level within coastal wetlands from a stack of consecutive interferograms acquired with very short temporal separation (~30 min). The algorithm is applied to six consecutive UAVSAR images collected in tidal wetlands of the Wax Lake Delta, Louisiana, USA. Validation of our water level change retrievals with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">in situ</i> field observations was conclusive with high correlation and an RMSE generally smaller than 3 cm. Comparison of our algorithm with other phase unwrapping error correction methods shows significant improvement (30%–35% increase in the number of correctly unwrapped pixels) when applied to rapid changes in water level. The set of corrections presented in this work enables measurement of water level change in deltas and other areas where tides drive highly dynamic flooding of inland vegetated areas. Although demonstrated for water level change, the method is applicable to other InSAR datasets with large spatial gradients or observed discontinuities between coherent but spatially isolated areas.

Motion Estimation of Underwater Platforms Using Impulse Responses From the Seafloor

Autonomous underwater vehicles require accurate navigation. Techniques such as image registration using consecutive acoustic images from a sonar have shown promising results for this task. The implementation of such techniques using sonar images augmented with deep learning (DL) networks demonstrate high navigation accuracy; this is possible even with highly compressed images. The sonar images are estimates of sampled in time (with a ping period) magnitudes of channel impulse responses representing the underwater acoustic environment. More information about the environment is contained in (almost) continuous in time estimates of the channel impulse responses. Such estimates can be obtained using full-duplex technology. Rather than using sonar images, this paper investigates the use of channel impulse response estimates for underwater platform motion estimation. The proposed system uses a single projector and a small number of receiving transducers installed on the moving platform. A DL network is used to estimate the motion in two degrees of freedom (forward/backward and sideways), using two or more consecutive impulse response estimates as the input. To train the DL network, a specially designed simulator is used to model the underwater acoustic environment, populated with multiple objects spread on the seafloor. The proposed technique can significantly reduce the acoustic hardware and processing complexity of the DL network and obtain a higher accuracy of motion estimation, compared with techniques based on the processing of sonar images, e.g., the error achieved with the technique proposed in this paper is 1.7% of the maximum platform displacement, compared to 4% achieved with a technique using sonar images. The navigation accuracy is further illustrated by examples of estimation of complex trajectories.

Convective Precipitation Nowcasting Using U-Net Model

Convective precipitation nowcasting remains challenging due to the fast change in convective weather. Radar images are the most important data source in nowcasting research area. This study proposes a radar data-based U-Net model for precipitation nowcasting. The nowcasting problem is first transformed into an image-to-image translation problem in deep learning under the U-Net architecture, which is based on convolutional neural networks (CNNs). The input of the model is five consecutive radar images; the output is the predicted radar reflectivity image. The model consists of three operations: upsampling, downsampling, and skip connection. Three methods, U-Net, TREC, and TrajGRU, are used for comparison in the experiments. The experimental results show that both deep learning methods outperform the TREC method, and the CNN-based U-Net can achieve almost the same performance as TrajGRU which is a recurrent neural network (RNN)-based model. With the advantages that U-Net is simple, efficient, easy to understand, and customize, this result shows the great potential of CNN-based models in addressing time-series applications.

Edge Tracing Using Gaussian Process Regression.

We introduce a novel edge tracing algorithm using Gaussian process regression. Our edge-based segmentation algorithm models an edge of interest using Gaussian process regression and iteratively searches the image for edge pixels in a recursive Bayesian scheme. This procedure combines local edge information from the image gradient and global structural information from posterior curves, sampled from the model's posterior predictive distribution, to sequentially build and refine an observation set of edge pixels. This accumulation of pixels converges the distribution to the edge of interest. Hyperparameters can be tuned by the user at initialisation and optimised given the refined observation set. This tunable approach does not require any prior training and is not restricted to any particular type of imaging domain. Due to the model's uncertainty quantification, the algorithm is robust to artefacts and occlusions which degrade the quality and continuity of edges in images. Our approach also has the ability to efficiently trace edges in image sequences by using previous-image edge traces as a priori information for consecutive images. Various applications to medical imaging and satellite imaging are used to validate the technique and comparisons are made with two commonly used edge tracing algorithms.

Learning Dense and Continuous Optical Flow From an Event Camera.

Event cameras such as DAVIS can simultaneously output high temporal resolution events and low frame-rate intensity images, which own great potential in capturing scene motion, such as optical flow estimation. Most of the existing optical flow estimation methods are based on two consecutive image frames and can only estimate discrete flow at a fixed time interval. Previous work has shown that continuous flow estimation can be achieved by changing the quantities or time intervals of events. However, they are difficult to estimate reliable dense flow, especially in the regions without any triggered events. In this paper, we propose a novel deep learning-based dense and continuous optical flow estimation framework from a single image with event streams, which facilitates the accurate perception of high-speed motion. Specifically, we first propose an event-image fusion and correlation module to effectively exploit the internal motion from two different modalities of data. Then we propose an iterative update network structure with bidirectional training for optical flow prediction. Therefore, our model can estimate reliable dense flow as two-frame-based methods, as well as estimate temporal continuous flow as event-based methods. Extensive experimental results on both synthetic and real captured datasets demonstrate that our model outperforms existing event-based state-of-the-art methods and our designed baselines for accurate dense and continuous optical flow estimation.