High-resolution Ones Research Articles

Enhancing image resolution of confocal fluorescence microscopy with deep learning

Super-resolution optical imaging is crucial to the study of cellular processes. Current super-resolution fluorescence microscopy is restricted by the need of special fluorophores or sophisticated optical systems, or long acquisition and computational times. In this work, we present a deep-learning-based super-resolution technique of confocal microscopy. We devise a two-channel attention network (TCAN), which takes advantage of both spatial representations and frequency contents to learn a more precise mapping from low-resolution images to high-resolution ones. This scheme is robust against changes in the pixel size and the imaging setup, enabling the optimal model to generalize to different fluorescence microscopy modalities unseen in the training set. Our algorithm is validated on diverse biological structures and dual-color confocal images of actin-microtubules, improving the resolution from ~ 230 nm to ~ 110 nm. Last but not least, we demonstrate live-cell super-resolution imaging by revealing the detailed structures and dynamic instability of microtubules.

Structure-Aware Deep Networks and Pixel-Level Generative Adversarial Training for Single Image Super-Resolution

The resolution of current display devices is getting higher and higher, and 4K/8K display devices have become popular, which requires image super-resolution technologies to enlarge and restore the input low-resolution images into high-resolution ones. In addition, image super-resolution technologies can facilitate various vision-based measurement applications. Recently, super-resolution methods based on generative adversarial networks (GAN) have become the mainstream. However, some recent studies shown that GAN-based image super-resolution methods will cause structural distortion. Existing methods alleviate the problem of structural distortion by enhancing structure generation. But this type of methods cannot essentially solve the problem of structural distortion caused by adversarial training. In contrast, this paper proposes the pixel-level generative adversarial training to solve the structural distortion problem, which finely constrains the structure of images during the adversarial training process. In addition, to better generate the structure and details of images and make full use of similar texture details within images, we build a structure-aware image super-resolution network, which not only enhances the structure generation through gradient guidance but also effectively integrates non-local self-similarity modules in a multi-level manner. Experimental results show that the proposed method achieves better quantitative and qualitative results than the state-of-the-art methods. The ablation experiments show that the proposed structure-aware deep network and pixel-level adversarial training can improve the performance of image super-resolution.

Enhanced measurements of structural inter-story drift responses in shaking table tests

Accurate measurements of inter-story drift responses are critical in shaking table tests. This paper compared three commonly-used approaches of inter-story drift measurement, and developed the techniques for enhanced measurement. This study proposed a novel arrangement of displacement meters along with the associated data correction method. By setting the overhanging steel arms above and beneath a floor slab at the same position, the suggested approach could remove the influence of floor slab rotation and thus improve the accuracy of inter-story drift measurement. In addition, a novel computer vision-based target tracking approach based on a super-resolution (SR) image reconstruction technique was developed. This advanced deep learning-based SR method can transform blurry, low-resolution images into sharp, high-resolution ones for precise target tracking. The accuracy of these developed inter-story measurement approaches was evaluated through a case study of shaking table tests of a large-scale three-story reinforced concrete (RC) building structure. The results indicated that the novel arrangement of displacement meters and associated data correction method successfully eliminated the influence of floor slab rotation, which could result in an error of approximately 20% in the inter-story drift measurement if left uncorrected. The novel SR method overcame the limitation of video resolution and achieved a stable sub-pixel measurement result. In the case of seismic loading, the SR method improved the signal-to-noise ratio of the drift measurement by 68%, and reduced the root mean square error by 63%, compared with the conventional template matching technique. The modal parameters of the test structure were accurately identified from the small-magnitude displacement data of white noise vibration responses measured using the SR method.

Long-term missing wind data recovery using free access databases and deep learning for bridge health monitoring

Wind environment monitoring at the bridge site is essential in structural health monitoring (SHM). However, as the performance of the electronic equipment for bridge SHM system deteriorates, wind monitoring data often suffer from long-term continuous data missing. The duration of data missing may be as long as several months, creating barriers to safety monitoring and monitoring data mining of the bridge structures. The conventional interpolation techniques hardly achieve the desired data recovery. Therefore, a framework was proposed for long-term missing wind data recovery based on a deep neural network (DNN) utilizing a free access database (European Center for Medium-Range Weather Forecasts, ECMWF). This framework consisted of one regression task (Task 1) and one temporal super-resolution task (Task 2). In Task 1, the hourly wind data provided by ECMWF were first learned to the hourly ones of the bridge SHM system. In Task 2, the low-resolution wind data (hourly averages) were upsampled to high-resolution ones (10-min averages). The U-net architecture provided the basis for the DNNs in both tasks. Unlike the conventional time-domain loss function used in Task 1, Task 2 adopted a time-frequency cross-domain loss function for training, which innovatively employed a spectrum magnitude balance strategy to enhance the reconstruction of the high-frequency components of wind speed signals. The proposed framework's feasibility and effectiveness were verified through a case study of recovering long-term continuous missing wind data in the SHM system of Sutong Bridge, China. The proposed methodology provides a new perspective for recovering long-term continuous missing SHM data.

Super-Resolution-Empowered Adaptive Medical Video Streaming in Telemedicine Systems

Due to influence of COVID-19, telemedicine is becoming more and more important. High-quality medical videos can provide a physician with a better visual experience and increase the accuracy of disease diagnosis, but this requires a dramatic increase in bandwidth compared to that required by regular videos. Existing adaptive video-streaming approaches cannot successfully provide high-resolution video-streaming services under poor or fluctuating network conditions with limited bandwidth. In this paper, we propose a super-resolution-empowered adaptive medical video streaming in telemedicine system (named SR-Telemedicine) to provide high quality of experience (QoE) videos for the physician while saving the network bandwidth. In SR-Telemedicine, very low-resolution video chunks are first transmitted from the patient to an edge computing node near the physician. Then, a video super-resolution (VSR) model is employed at the edge to reconstruct the low-resolution video chunks into high-resolution ones with an appropriate high-resolution level (such as 720p or 1080p). Furthermore, the neural network of VSR model is designed to be scalable and can be determined dynamically. Based on the time-varying computational capability of the edge computing node and the network condition, a double deep Q-Network (DDQN)-based algorithm is proposed to jointly select the optimal reconstructed high-resolution level and the scale of the VSR model. Finally, extensive experiments based on real-world traces are carried out, and the experimental results illustrate that the proposed SR-Telemedicine system can improve the QoE of medical videos by 17–79% compared to three baseline algorithms.

An Iteratively Optimized Patch Label Inference Network for Automatic Pavement Distress Detection

We present a novel deep learning framework named the Iteratively Optimized Patch Label Inference Network (IOPLIN) for automatically detecting various pavement distresses that are not solely limited to specific ones, such as cracks and potholes. IOPLIN can be iteratively trained with only the image label via the Expectation-Maximization Inspired Patch Label Distillation (EMIPLD) strategy, and accomplish this task well by inferring the labels of patches from the pavement images. IOPLIN enjoys many desirable properties over the state-of-the-art single branch CNN models such as GoogLeNet and EfficientNet. It is able to handle images in different resolutions, and sufficiently utilize image information particularly for the high-resolution ones, since IOPLIN extracts the visual features from unrevised image patches instead of the resized entire image. Moreover, it can roughly localize the pavement distress without using any prior localization information in the training phase. In order to better evaluate the effectiveness of our method in practice, we construct a large-scale Bituminous Pavement Disease Detection dataset named CQU-BPDD consisting of 60,059 high-resolution pavement images, which are acquired from different areas at different times. Extensive results on this dataset demonstrate the superiority of IOPLIN over the state-of-the-art image classification approaches in automatic pavement distress detection. The source codes of IOPLIN are released on <uri>https://github.com/DearCaat/ioplin</uri>, and the CQU-BPDD dataset is able to be accessed on <uri>https://dearcaat.github.io/CQU-BPDD/</uri>.

Deep learning-based single image super-resolution for low-field MR brain images

Low-field MRI scanners are significantly less expensive than their high-field counterparts, which gives them the potential to make MRI technology more accessible all around the world. In general, images acquired using low-field MRI scanners tend to be of a relatively low resolution, as signal-to-noise ratios are lower. The aim of this work is to improve the resolution of these images. To this end, we present a deep learning-based approach to transform low-resolution low-field MR images into high-resolution ones. A convolutional neural network was trained to carry out single image super-resolution reconstruction using pairs of noisy low-resolution images and their noise-free high-resolution counterparts, which were obtained from the publicly available NYU fastMRI database. This network was subsequently applied to noisy images acquired using a low-field MRI scanner. The trained convolutional network yielded sharp super-resolution images in which most of the high-frequency components were recovered. In conclusion, we showed that a deep learning-based approach has great potential when it comes to increasing the resolution of low-field MR images.

Unsupervised Super Resolution Network for Hyperspectral Histologic Imaging.

Hyperspectral imaging (HSI) has many advantages in microscopic applications, including high sensitivity and specificity for cancer detection on histological slides. However, acquiring hyperspectral images of a whole slide with a high image resolution and a high image quality can take a long scanning time and require a very large data storage. One potential solution is to acquire and save low-resolution hyperspectral images and reconstruct the high-resolution ones only when needed. The purpose of this study is to develop a simple yet effective unsupervised super resolution network for hyperspectral histologic imaging with the guidance of RGB digital histology images. High-resolution hyperspectral images of hemoxylin & eosin (H&E) stained slides were obtained at 10× magnification and down-sampled 2×, 4×, and 5× to generate low-resolution hyperspectral data. High-resolution digital histologic RGB images of the same field of view (FOV) were cropped and registered to the corresponding high-resolution hyperspectral images. A neural network based on a modified U-Net architecture, which takes the low-resolution hyperspectral images and high-resolution RGB images as inputs, was trained with unsupervised methods to output high-resolution hyperspectral data. The generated high-resolution hyperspectral images have similar spectral signatures and improved image contrast than the original high-resolution hyperspectral images, which indicates that the super resolution network with RGB guidance can improve the image quality. The proposed method can reduce the acquisition time and save storage space taken up by hyperspectral images without compromising image quality, which will potentially promote the use of hyperspectral imaging technology in digital pathology and many other clinical applications.

Video super-resolution with inverse recurrent net and hybrid local fusion

Video super-resolution converts low-resolution videos to sharp high-resolution ones. In order to make better use of temporal information in video super-resolution, we design inverse recurrent net and hybrid local fusion. We concatenate the original low-resolution input sequence and its inverse sequence repeatedly. The new sequence is viewed as a combination of different stages, and is processed sequentially by using orent net. The outputs of the last two stages in opposite directions are fused to generate the final images. Our inverse recurrent net can extract more bidirectional temporal information in the input sequence, without adding parameter to the corresponding unidirectional recurrent net. We also propose a hybrid local fusion method which uses parallel fusion and cascade fusion for incorporating sliding-window-based methods into our inverse recurrent net. Extensive experimental results demonstrate the effectiveness of the proposed inverse recurrent net and hybrid local fusion, in terms of visual quality and quantitative evaluations. The code will be released athttps://github.com/5ofwind.

Low-Resolution Infrared Array Sensor for Counting and Localizing People Indoors: When Low End Technology Meets Cutting Edge Deep Learning Techniques

In this paper, we propose a method that uses low-resolution infrared (IR) array sensors to identify the presence and location of people indoors. In the first step, we introduce a method that uses 32 × 24 pixels IR array sensors and relies on deep learning to detect the presence and location of up to three people with an accuracy reaching 97.84%. The approach detects the presence of a single person with an accuracy equal to 100%. In the second step, we use lower end IR array sensors with even lower resolution (16 × 12 and 8 × 6) to perform the same tasks. We invoke super resolution and denoising techniques to faithfully upscale the low-resolution images into higher resolution ones. We then perform classification tasks and identify the number of people and their locations. Our experiments show that it is possible to detect up to three people and a single person with accuracy equal to 94.90 and 99.85%, respectively, when using frames of size 16 × 12. For frames of size 8 × 6, the accuracy reaches 86.79 and 97.59%, respectively. Compared to a much complex network (i.e., RetinaNet), our method presents an improvement of over 8% in detection.

Video Satellite Imagery Super-Resolution via Model-Based Deep Neural Networks

Video satellite imagery has become a hot research topic in Earth observation due to its ability to capture dynamic information. However, its high temporal resolution comes at the expense of spatial resolution. In recent years, deep learning (DL) based super-resolution (SR) methods have played an essential role to improve the spatial resolution of video satellite images. Instead of fully considering the degradation process, most existing DL-based methods attempt to learn the relationship between low-resolution (LR) satellite video frames and their corresponding high-resolution (HR) ones. In this paper, we propose model-based deep neural networks for video satellite imagery SR (VSSR). The VSSR is composed of three main modules: degradation estimation module, intermediate image generation module, and multi-frame feature fusion module. First, the blur kernel and noise level of LR video frames are flexibly estimated by the degradation estimation module. Second, an intermediate image generation module is proposed to iteratively solve two optimal subproblems and the outputs of this module are intermediate SR frames. Third, a three-dimensional (3D) feature fusion subnetwork is leveraged to fuse the features from multiple video frames. Different from previous video satellite SR methods, the proposed VSSR is a multi-frame-based method that can merge the advantages of both learning-based and model-based methods. Experiments on real-world Jilin-1 and OVS-1 video satellite images have been conducted and the SR results demonstrate that the proposed VSSR achieves superior visual effects and quantitative performance compared with the state-of-the-art methods.

Enhancing SRTM digital elevation models with deep-learning-based super-resolution image generation

Digital elevation models are responsible for providing altimetric information on a surface to be mapped. While global models of low and medium spatial resolution are available open source by several space agencies, the high-resolution ones, which are utilized in scales 1:25,000 and larger, are scarce and expensive. Here we address this limitation by the utilization of deep learning algorithms coupled with SISR techniques in digital elevation models to obtain better spatial quality versions from lower resolution inputs. The development of a GAN-based methodology enables the improvement of the initial spatial resolution of low-resolution images. A dataset with different pairs of digital elevation models was created with the objective of allowing the study to be carried out, promoting the emergence of new research groups in the area as well as enabling the comparison between the results obtained. It has been found that by increasing the number of iterations the performance of the generated model was improved and the quality of the generated image increased. Furthermore, the visual analysis of the generated image against the high- and low-resolution ones showed a great similarity between the first two.

Feature Map Distillation of Thin Nets for Low-Resolution Object Recognition.

Intelligent video surveillance is an important computer vision application in natural environments. Since detected objects under surveillance are usually low-resolution and noisy, their accurate recognition represents a huge challenge. Knowledge distillation is an effective method to deal with it, but existing related work usually focuses on reducing the channel count of a student network, not feature map size. As a result, they cannot transfer "privilege information" hidden in feature maps of a wide and deep teacher network into a thin and shallow student one, leading to the latter's poor performance. To address this issue, we propose a Feature Map Distillation (FMD) framework under which the feature map size of teacher and student networks is different. FMD consists of two main components: Feature Decoder Distillation (FDD) and Feature Map Consistency-enforcement (FMC). FDD reconstructs the shallow texture features of a thin student network to approximate the corresponding samples in a teacher network, which allows the high-resolution ones to directly guide the learning of the shallow features of the student network. FMC makes the size and direction of each deep feature map consistent between student and teacher networks, which constrains each pair of feature maps to produce the same feature distribution. FDD and FMC allow a thin student network to learn rich "privilege information" in feature maps of a wide teacher network. The overall performance of FMD is verified in multiple recognition tasks by comparing it with state-of-the-art knowledge distillation methods on low-resolution and noisy objects.

High-fidelity reconstruction of turbulent flow from spatially limited data using enhanced super-resolution generative adversarial network

In this study, a deep learning-based approach is applied with the aim of reconstructing high-resolution turbulent flow fields using minimal flow field data. A multi-scale enhanced super-resolution generative adversarial network with a physics-based loss function is introduced as a model to reconstruct the high-resolution flow fields. The model capability to reconstruct high-resolution laminar flows is examined using direct numerical simulation data of laminar flow around a square cylinder. The results reveal that the model can accurately reproduce the high-resolution flow fields even when limited spatial information is provided. The DNS data of turbulent channel flow at two friction Reynolds numbers Reτ=180 and 550 are used to assess the ability of the model to reconstruct the high-resolution wall-bounded turbulent flow fields. The instantaneous and statistical results obtained from the model agree well with the ground truth data, indicating that the model can successfully learn to map the coarse flow fields to the high-resolution ones. Furthermore, the possibility of performing transfer learning for the case of turbulent channel flow is thoroughly examined. The results indicate that the amount of the training data and the time required for training can be effectively reduced without affecting the performance of the model. The computational cost of the proposed model is also found to be effectively low. These results demonstrate that using high-fidelity training data with physics-guided generative adversarial network-based models can be practically efficient in reconstructing high-resolution turbulent flow fields from extremely coarse data.

RESOLUTION ENHANCEMENT OF INFRARED THERMAL IMAGING BY PANSHARPENING ALGORITHMS

Abstract. One common tool in Cultural Heritage inspections is thermal cameras, which are sensitive to the infrared part of the electromagnetic spectrum. But the resolution of these sensors is quite lower than other kinds like visible spectrum range cameras. Typically, the sensors in thermal cameras do not exceed the megapixel frontier. This limitation becomes a problem when trying to combine the information from the thermal images with data from other sensors with much higher resolution such as visible RGB cameras in the same project.In Remote Sensing, algorithms have been designed to fuse multispectral images with panchromatic images (in origin from satellite platforms) to enhance the resolution of lower resolution images with higher resolution ones. These processes are known as pansharpening. Although pansharpening procedures are widely known, they have not been tested working with thermal imaging. The first approach of merging thermal and visual spectrum images to enhance the resolution of the original thermal image involved applying the intensity-hue-saturation (IHS) algorithm (Lagüela et al., 2012, Kuenzer and Dech, 2013). These works only studied one particular algorithm and they did not include any quality study of the results.Our work contains a complete review of a bigger pansharpening algorithms’ set and provides an in-depth study of thermal imaging pansharpening, with a numerical assessment. Our research allows the use of thermal sensors with a lower resolution than other types of sensors used simultaneously in the same project.

Herbig Ae/Be Stars toward the Dark Cloud LDN 1667*

Abstract We report the discovery of a new emission-line object, named SPH 4-South = (GAIA EDR3 5616553300192230272), toward the dark cloud LDN 1667. This object came to our attention after inspecting public images that show a faint diffuse nebula a few arcseconds south of SPH 4, an emission-line object previously classified as a T Tauri star. We present high-resolution spectra and analyzed JHK photometry of SPH 4 and SPH 4-South and new narrowband and archival broadband images of these objects. A comparison of the spectra of SPH 4 and SPH 4-South with high-resolution ones of DG Cir and R Mon strongly suggests that SPH 4 and SPH 4-South are Herbig Ae/Be stars. The classification of SPH 4-South is further supported by using a k-NN algorithm to its position in an H−K versus J−H color–color diagram. Both stars are detected in the four WISE bands and the WISE colors allow us to classify SPH 4 as a Class I and SPH 4-South as a Class II source. We also show that the faint nebula is most probably associated with SPH 4-South. Using published results on LDN 1667 and the Gaia Early Data Release 3, we conclude that SPH 4 is a member of LDN 1667. The case of SPH 4-South is not clear because the determination of its distance and proper motion could be affected by the nebulosity around the star, although membership of SPH 4-South to LDN 1667 cannot be ruled out.

Image Degradation Modeling for Real-World Super Resolution via Conditional Normalizing Flow

In recent years, deep-learning-based super-resolution (SR) methods have obtained impressive performance gains on synthetic clean datasets, but they fail to perform well in real-world scenarios due to insufficient real-world training data. To tackle this issue, we propose a conditional-normalizing-flow-based method named IDFlow for image degradation modeling that aims to generate various degraded low-resolution (LR) images for real-world SR model training. IDFlow takes image degradation modeling as a problem of learning a conditional probability distribution of LR images given the high-resolution (HR) ones, and learns the distribution from existing real-world SR datasets. It first decomposes the image degradation modeling into blur degradation modeling and real-world noise modeling. It then utilizes two multi-scale invertible networks to model these two steps, respectively. Before applied into real-world SR, IDFlow is first trained supervisedly on two real-world datasets RealSR and DPED with negative log-likelihood (NLL) loss. It is then used to generate a large number of HR-LR image pairs from an arbitrary HR image dataset for SR model training. Extensive experiments on IDFlow with RealSR and DPED are conducted, including evaluations on image degradation stochasticity, degradation modeling, and real-world super resolution. Two known SR models are trained with IDFlow and named as IDFlow-SR and IDFlow-GAN. Testing results on the RealSR and DPED testing dataset show that not only can IDFlow generate realistic degraded images close to real-world images, but it is also beneficial to real-world SR performance improvement. IDFlow-SR achieves 4× SR performance gains of 0.91 dB and 0.161 in terms of image quality assessment metrics PSNR and LPIPS. Moreover, IDFlow-GAN can super-resolve real-world images in the DPED testing dataset with richer textures and maintain clearer patterns without visible noises when compared with state-of-the-art SR methods. Quantitative and qualitative experimental results well demonstrate the effectiveness of the proposed IDFlow.

Skill in Stone Knapping: an Ecological Approach

Skill has allowed lithic analyses to expand their scope beyond the limits set by a representational understanding of practices and sociocultural dynamics. It remains excluded from coarse archaeological contexts in favour of higher resolution ones however. Such coarse contexts are ubiquitous and must be included to broaden description, interpretation and theorization into broader and more heterogeneous narrative landscapes. This paper argues that skill is key to including lithic practices from coarser archaeological palimpsests, provided it is reframed as a process immanent to any cultural practice that conjoins with other processes to shape contexts of various scales. Second, skill must be anchored with a set of core concepts—technical difficulty, accidents and execution quality—that each knapping event and every lithic assemblage actualizes, regardless of scale. Third, methodologies must be built using this set of core concepts and adapted to a site’s specifics. Using such a methodology, I describe learning patterns, skilled reduction sequences and spatial patterning in the plowed fields of La Martre (Quebec, Canada), where millennia of continuous occupation and hundreds of thousands of lithic remains have been mixed up in a dense and homogeneous layer. I show that understanding skill as a trans-scalar process can help free lithic analyses from prior, bounded and familiar units of analysis. It can and should be used first to draw broader patterns that connect contextually specific lithic expressions. It affords for scalable analyses that can help expand the scope of the depositional contexts archaeologists routinely work with.

Enlighten-GAN for Super Resolution Reconstruction in Mid-Resolution Remote Sensing Images

Previously, generative adversarial networks (GAN) have been widely applied on super resolution reconstruction (SRR) methods, which turn low-resolution (LR) images into high-resolution (HR) ones. However, as these methods recover high frequency information with what they observed from the other images, they tend to produce artifacts when processing unfamiliar images. Optical satellite remote sensing images are of a far more complicated scene than natural images. Therefore, applying the previous networks on remote sensing images, especially mid-resolution ones, leads to unstable convergence and thus unpleasing artifacts. In this paper, we propose Enlighten-GAN for SRR tasks on large-size optical mid-resolution remote sensing images. Specifically, we design the enlighten blocks to induce network converging to a reliable point, and bring the Self-Supervised Hierarchical Perceptual Loss to attain performance improvement overpassing the other loss functions. Furthermore, limited by memory, large-scale images need to be cropped into patches to get through the network separately. To merge the reconstructed patches into a whole, we employ the internal inconsistency loss and cropping-and-clipping strategy, to avoid the seam line. Experiment results certify that Enlighten-GAN outperforms the state-of-the-art methods in terms of gradient similarity metric (GSM) on mid-resolution Sentinel-2 remote sensing images.

WITHDRAWN: Super resolution using generative adversarial networks

Abstract Resolution has continually been a problem since the term ‘picture’ or ‘image’ came into existence. There have been many researches on improvising the resolution of an image however none could create very fast sufficient strategies to get the correct effects that can manage both time and production of best outputs. So, in this, we are going to discover about super resolution which has been a hassle for many years with the usage of signal processing strategies. But now we are able to clear up this trouble using deep learning strategies. The method we endorse here can flip a low-resolution photo/video to high resolution ones, up to a pleasant and exceptional level. We aim to obtain these using GANs (Generative Adversarial Networks) at considerable upscaling factors.