Single Image Research Articles

G2NPAN: GAN-guided nuance perceptual attention network for multimodal medical fusion image quality assessment

Multimodal medical fusion images (MMFI) are formed by fusing medical images of two or more modalities with the aim of displaying as much valuable information as possible in a single image. However, due to the different strategies of various fusion algorithms, the quality of the generated fused images is uneven. Thus, an effective blind image quality assessment (BIQA) method is urgently required. The challenge of MMFI quality assessment is to enable the network to perceive the nuances between fused images of different qualities, and the key point for the success of BIQA is the availability of valid reference information. To this end, this work proposes a generative adversarial network (GAN) -guided nuance perceptual attention network (G2NPAN) to implement BIQA for MMFI. Specifically, we achieve the blind evaluation style via the design of a GAN and develop a Unique Feature Warehouse module to learn the effective features of fused images from the pixel level. The redesigned loss function guides the network to perceive the image quality. In the end, the class activation mapping supervised quality assessment network is employed to obtain the MMFI quality score. Extensive experiments and validation have been conducted in a database of medical fusion images, and the proposed method is superior to the state-of-the-art BIQA method.

Light Field Display Point Rendering

Rendering for light field displays (LFDs) requires rendering of dozens or hundreds of views, which must then be combined into a single image on the display, making real-time LFD rendering extremely difficult. We introduce light field display point rendering (LFDPR), which meets these challenges by improving eye-based point rendering [Gavane and Watson 2023] with texture-based splatting, which avoids oversampling of triangles mapped to only a few texels; and with LFD-biased sampling, which adjusts horizontal and vertical triangle sampling to match the sampling of the LFD itself. To improve image quality, we introduce multiview mipmapping, which reduces texture aliasing even though compute shaders do not support hardware mipmapping. We also introduce angular supersampling and reconstruction to combat LFD view aliasing and crosstalk. The resulting LFDPR is 2-8x times faster than multiview rendering, with similar comparable quality.

Improved 3D human face reconstruction from 2D images using blended hard edges

AbstractThis study reports an effective and robust edge-based scheme for the reconstruction of 3D human faces from input of single images, addressing drawbacks of existing methods in case of large face pose angles or noisy input images. Accurate 3D face reconstruction from 2D images is important, as it can enable a wide range of applications, such as face recognition, animations, games and AR/VR systems. Edge features extracted from 2D images contain wealthy and robust 3D geometric information, which were used together with landmarks for face reconstruction purpose. However, the accurate reconstruction of 3D faces from contour features is a challenging task, since traditional edge or contour detection algorithms introduce a great deal of noise, which would adversely affect the reconstruction. This paper reports on the use of a hard-blended face contour feature from a neural network and a Canny edge extractor for face reconstruction. The quantitative results indicate that our method achieves a notable improvement in face reconstruction with a Euclidean distance error of 1.64 mm and a normal vector distance error of 1.27 mm when compared to the ground truth, outperforming both traditional and other deep learning-based methods. These metrics show particularly significant advancements, especially in face shape reconstruction under large pose angles. The method also achieved higher accuracy and robustness on in-the-wild images under conditions of blurring, makeup, occlusion and poor illumination.

A Novel Lightweight Model for Underwater Image Enhancement

Underwater images suffer from low contrast and color distortion. In order to improve the quality of underwater images and reduce storage and computational resources, this paper proposes a lightweight model Rep-UWnet to enhance underwater images. The model consists of a fully connected convolutional network and three densely connected RepConv blocks in series, with the input images connected to the output of each block with a Skip connection. First, the original underwater image is subjected to feature extraction by the SimSPPF module and is processed through feature summation with the original one to be produced as the input image. Then, the first convolutional layer with a kernel size of 3 × 3, generates 64 feature maps, and the multi-scale hybrid convolutional attention module enhances the useful features by reweighting the features of different channels. Second, three RepConv blocks are connected to reduce the number of parameters in extracting features and increase the test speed. Finally, a convolutional layer with 3 kernels generates enhanced underwater images. Our method reduces the number of parameters from 2.7 M to 0.45 M (around 83% reduction) but outperforms state-of-the-art algorithms by extensive experiments. Furthermore, we demonstrate our Rep-UWnet effectively improves high-level vision tasks like edge detection and single image depth estimation. This method not only surpasses the contrast method in objective quality, but also significantly improves the contrast, colorimetry, and clarity of underwater images in subjective quality.

Single-Image Building Height Estimation Using EfficientNet: A Simplified, Scalable Approach

We present a novel approach for estimating building heights using single street-level images. The method employs EfficientNet, a state-of-the-art neural network, to eliminate the need for additional data like street maps. We compare this new method with existing techniques, focusing on accuracy evaluated through metrics like Mean Absolute Error (MAE). The model is pre-trained on the Cityscapes dataset and fine-tuned on images from Toronto’s 3D Massing dataset. It demonstrates strong accuracy, with an MAE of 1.21 meters, outperforming traditional methods.

Fusion of full-field optical angiography images via gradient feature detection

Full-field optical angiography (FFOA)—a real-time non-invasive imaging technique for extracting biological blood microcirculation information—contributes to an in-depth understanding of the functional and pathological changes of biological tissues. However, owing to the limitation of the depth-of-field (DOF) of optical lenses, existing FFOA imaging methods cannot capture an image containing every blood-flow information. To address this problem, this study develops a long-DOF full-field optical angiography imaging system and proposes a novel multi-focus image fusion scheme to expand the DOF. First, FFOA images with different focal lengths are acquired by the absorption intensity fluctuation modulation effect. Second, an image fusion scheme based on gradient feature detection in a nonsubsampled contourlet transform domain is developed to capture focus features from FFOA images and synthesize an all-focused image. Specifically, FFOA images are decomposed by NSCT into coefficients and low-frequency difference images; thereafter, two gradient feature detection-based fusion rules are used to select the pre-fused coefficients. The experimental results of both phantom and animal cases show that the proposed fusion method can effectively extend the DOF and address practical FFOA image defocusing problems. The fused FFOA image can provide a more comprehensive description of blood information than a single FFOA image.

Measuring the Optical Concurrence of Vector Beams with an Atomic-State Interferometer

We investigate the transmission of vector beams, correlated in their polarization and spatial degrees of freedom, through cold atoms in the presence of a transverse magnetic coupling field. The resulting phase-dependent dynamics allow us to imprint the spatially varying polarization of a vector beam onto atomic spin polarizations, thereby establishing a direct link between optical space-polarization correlations and atomic-state interference. We find that the resulting absorption profiles show interference fringes whose modulation strength is given by the squared concurrence of the vector beam, letting us identify optical concurrence from a single absorption image. We expect impact across a diverse range of applications, including spintronics, quantum memories, metrology, and clocks. Published by the American Physical Society 2024

A 600 m2 array of 6.5 m telescopes at the lunar pole.

The proposed lunar telescope for optical and infrared astronomy aims at very large aperture, 600 m2, at a fundable cost. It comprises an array of 18 separate telescopes, each of 6.5 m aperture. The 200 m diameter array will be located within 1/2° (15 km) of a lunar pole on approximately level ground, with a perimeter screen deployed to provide shade and cooling to cryogenic temperature. The 500 m diameter screen will allow unobscured access down to 8° elevation. All 18 telescopes will reflect light into a central beam combiner to form a single image covering wavelengths from 0.4 µm to 10 µm. The initial instrument complement will include high-resolution and multi-object spectrographs to exploit the single combined field of view of two arcminute diameter, with the diffraction limited resolution of 6.5 m aperture. Scientific applications include the search for molecular biosignatures in transiting exoplanets, and the study of galaxy evolution using red-shifted spectra to beyond z = 10. The array cost, including delivery to the Moon by SpaceX Starship for installation using lunar base infrastructure, is around $10 billion, similar to that of the 25 m2 JWST. To test the concept, first a single prototype 6.5 m unit would be operated at the lunar south pole. This article is part of a discussion meeting issue 'Astronomy from the Moon: the next decades (part 2)'.

Self-Supervised Monocular Depth Estimation via Binocular Geometric Correlation Learning

Monocular depth estimation aims to infer a depth map from a single image. Although supervised learning-based methods have achieved remarkable performance, they generally rely on a large amount of labor-intensively annotated data. Self-supervised methods on the other hand do not require any annotation of ground-truth depth and have recently attracted increasing attention. In this work, we propose a self-supervised monocular depth estimation network via binocular geometric correlation learning. Specifically, considering the inter-view geometric correlation, a binocular cue prediction module is presented to generate the auxiliary vision cue for the self-supervised learning of monocular depth estimation. Then, to deal with the occlusion in depth estimation, an occlusion interference attenuated constraint is developed to guide the supervision of the network by inferring the occlusion region and producing paired occlusion masks. Experimental results on two popular benchmark datasets have demonstrated that the proposed network obtains competitive results compared to state-of-the-art self-supervised methods and achieves comparable results to some popular supervised methods.

Extracting the winter wheat using the decision tree based on time series dual-polarization SAR feature and NDVI.

Winter wheat is one of the most important crops in the world. It is great significance to obtain the planting area of winter wheat timely and accurately for formulating agricultural policies. Due to the limited resolution of single SAR data and the susceptibility of single optical data to weather conditions, it is difficult to accurately obtain the planting area of winter wheat using only SAR or optical data. To solve the problem of low accuracy of winter wheat extraction only using optical or SAR images, a decision tree classification method combining time series SAR backscattering feature and NDVI (Normalized Difference Vegetation Index) was constructed in this paper. By synergy using of SAR and optical data can compensate for their respective shortcomings. First, winter wheat was distinguished from other vegetation by NDVI at the maturity stage, and then it was extracted by SAR backscattering feature. This approach facilitates the semi-automated extraction of winter wheat. Taking Yucheng City of Shandong Province as study area, 9 Sentinel-1 images and one Sentinel-2 image were taken as the data sources, and the spatial distribution of winter wheat in 2022 was obtained. The results indicate that the overall accuracy (OA) and kappa coefficient (Kappa) of the proposed method are 96.10% and 0.94, respectively. Compared with the supervised classification of multi-temporal composite pseudocolor image and single Sentinel-2 image using Support Vector Machine (SVM) classifier, the OA are improved by 10.69% and 5.66%, respectively. Compared with using only SAR feature for decision tree classification, the producer accuracy (PA) and user accuracy (UA) for extracting the winter wheat are improved by 3.08% and 8.25%, respectively. The method proposed in this paper is rapid and accurate, and provide a new technical method for extracting winter wheat.

End-to-End lightweight Transformer-Based neural network for grasp detection towards fruit robotic handling

Robotic picking and placing are common operations for fruits and vegetables in grading, sorting or packaging systems. However, due to the diverse shapes and irregular surfaces of fruits and vegetables, improper handling during the picking process can result in detachment or damage. To ensure the correct grasping positions, it is necessary to design targeted neural network algorithms for achieving intelligent sorting. Therefore, this study focuses on 20 common fruit and vegetable agricultural products to develop a deep learning-based grasping detection algorithm model. By combining local features from convolutional neural networks with global features from Transformers, a lightweight end-to-end fruit and vegetable grasping detection network, MDETR, is constructed. Experimental results demonstrate that the MDETR algorithm not only achieves high accuracy in fruit and vegetable grasping detection but also improves the speed of pose detection. The average time required for detecting a single image is approximately 29.6 ms, meeting real-time requirements. The algorithm achieves a pose detection accuracy rate of 96 %, enabling precise detection and positioning of fruit and vegetable poses and achieving fast and accurate picking and placing. Additionally, a Pybullet simulation platform is developed for conducting grasping experiments, where the MDETR model achieves a grasping success rate of 88.9 %. This validates the robustness and generalization capabilities of the proposed detection algorithm model, designed specifically for fruit and vegetable grasping tasks.

Multi-scale network for single image deblurring based on ensemble learning module

Multi-scale network for single image deblurring based on ensemble learning module

LEAPSE: Learning Environment Affordances for 3D Human Pose and Shape Estimation.

We live in a 3D world where people interact with each other in the environment. Learning 3D posed humans therefore requires us to perceive and interpret these interactions. This paper proposes LEAPSE, a novel method that learns salient instance affordances for estimating a posed body from a single RGB image in a non-parametric manner. Existing methods mostly ignore the environment and estimate the human body independently from the surroundings. We capture the influences of non-contact and contact instances on a posed body as an adequate representation of the "environment affordances". The proposed method learns the global relationships between 3D joints, body mesh vertices, and salient instances as environment affordances on the human body. LEAPSE achieved state-of-the-art results on the 3DPW dataset with many affordance instances, and also demonstrated excellent performance on Human3.6M dataset. We further demonstrate the benefit of our method by showing that the performance of existing weak models can be significantly improved when combined with our environment affordance module.

Semi-supervised single image dehazing based on dual-teacher-student network with knowledge transfer

Semi-supervised single image dehazing based on dual-teacher-student network with knowledge transfer

Low Bone Mineral Density on Computed Tomography: Association with Poor Survival after Transcatheter Aortic Valve Replacement.

Background: Transcatheter aortic valve replacement (TAVR) has evolved as first-line therapy for severe aortic valve stenosis (AS), with pre-procedural computed tomography (CT) providing critical anatomical information. While primarily used for anatomical planning, TAVR-CT also offers an opportunity to assess low bone mineral density (BMD), a known indicator of frailty. Despite this, the prognostic role of BMD in TAVR patients remains unknown. This study aimed to evaluate BMD on routine TAVR-CT and its impact on long-term survival. Methods: In this retrospective study, 770 consecutive TAVR patients (mean age 80.7 ± 6.7 years, 54.0% males) between November 2015 and March 2022 were included. BMD was measured from a single axial image at the thoracic vertebral level on unenhanced CT scans. Cox regression models assessed the impact of BMD on mortality, and Restricted Cubic Spline models identified potential mortality thresholds. Results: The mean BMD value, as measured on non-contrast CT, was 147.5 ± 5.4 Hounsfield units, demonstrating a noteworthy association with mortality (adjusted hazard ratio per 100 HU decrease: 1.27 [95%CI: 1.01-1.59], p = 0.041). Restricted cubic spline analysis indicated that BMD below 200 HU was linked to a substantial increase in mortality risk. Upon crude Cox regression analysis, every 100 HU decrease was associated with a 32% increase in risk for death (HR 1.32 [95%CI: 1.068-1.65)], p = 0.010). Conclusions: In conclusion, low BMD on TAVR-CT is independently associated with reduced survival, suggesting its potential as a tool for comprehensive frailty assessment and improved risk prediction in TAVR patients.

Stereo matching from monocular images using feature consistency

AbstractSynthetic images facilitate stereo matching. However, synthetic images may suffer from image distortion, domain bias, and stereo mismatch, which would significantly restrict the widespread use of stereo matching models in the real world. The first goal in this paper is to synthesize real‐looking images for minimizing the domain bias between the synthesized and real images. For this purpose, sharpened disparity maps are produced from a mono real image. Then, stereo image pairs are synthesized using these imperfect disparity maps and the single real image in the proposed pipeline. Although the synthesized images are as realistic as possible, the domain styles of the synthesized images are always very different from the real images. Thus, the second goal is to enhance the domain generalization ability of the stereo matching network. For that, the feature extraction layer is replaced with a teacher–student model. Then, a constraint of binocular contrast features is imposed on the output of the model. When tested on the KITTI, ETH3D, and Middlebury datasets, the accuracy of the method outperforms traditional methods by at least 30%. Experiments demonstrate that the approaches are general and can be conveniently embedded into existing stereo networks.

Automated Medication Verification System (AMVS): System Based on Edge Detection and CNN Classification Drug on Embedded Systems

A novel automated medication verification system (AMVS) aims to address the limitation of manual medication verification among healthcare professionals with a high workload, thereby reducing medication errors in hospitals. Specifically, the manual medication verification process is time-consuming and prone to errors, especially in healthcare settings with high workloads. The proposed system strategy is to streamline and automate this process, enhancing efficiency and reducing medication errors. The system employs deep learning models to swiftly and accurately classify multiple medications within a single image without requiring manual labeling during model construction. It comprises edge detection and classification to verify medication types. Unlike previous studies conducted in open spaces, our study takes place in a closed space to minimize the impact of optical changes on image capture. During the experimental process, the system individually identifies each drug within the image by edge detection method and utilizes a classification model to determine each drug type. Our research has successfully developed a fully automated drug recognition system, achieving an accuracy of over 95% in identifying drug types and conducting segmentation analyses. Specifically, the system demonstrates an accuracy rate of approximately 96% for drug sets containing fewer than ten types and 93% for those with ten types. This verification system builds an image classification model quickly. It holds promising potential in assisting nursing staff during AMVS, thereby reducing the likelihood of medication errors and alleviating the burden on nursing staff.

A Multi-Level Relation-Aware Transformer model for occluded person re-identification

Occluded person re-identification (Re-ID) is a challenging task, as pedestrians are often obstructed by various occlusions, such as non-pedestrian objects or non-target pedestrians. Previous methods have heavily relied on auxiliary models to obtain information in unoccluded regions, such as human pose estimation. However, these auxiliary models fall short in accounting for pedestrian occlusions, thereby leading to potential misrepresentations. In addition, some previous works learned feature representations from single images, ignoring the potential relations among samples. To address these issues, this paper introduces a Multi-Level Relation-Aware Transformer (MLRAT) model for occluded person Re-ID. This model mainly encompasses two novel modules: Patch-Level Relation-Aware (PLRA) and Sample-Level Relation-Aware (SLRA). PLRA learns fine-grained local features by modeling the structural relations between key patches, bypassing the dependency on auxiliary models. It adopts a model-free method to select key patches that have high semantic correlation with the final pedestrian representation. In particular, to alleviate the interference of occlusion, PLRA captures the structural relations among key patches via a two-layer Graph Convolution Network (GCN), effectively guiding the local feature fusion and learning. SLRA is designed to facilitate the model to learn discriminative features by modeling the relations among samples. Specifically, to mitigate noisy relations of irrelevant samples, we present a Relation-Aware Transformer (RAT) block to capture the relations among neighbors. Furthermore, to bridge the gap between training and testing phases, a self-distillation method is employed to transfer the sample-level relations captured by SLRA to the backbone. Extensive experiments are conducted on four occluded datasets, two partial datasets and two holistic datasets. The results show that the proposed MLRAT model significantly outperforms existing baselines on four occluded datasets, while maintains top performance on two partial datasets and two holistic datasets.

Synthetic, non-person related panoramic radiographs created by generative adversarial networks in research, clinical, and teaching applications

ObjectivesGenerative Adversarial Networks (GANs) can produce synthetic images free from personal data. They hold significant value in medical research, where data protection is increasingly regulated. Panoramic radiographs (PRs) are a well-suited modality due to their significant level of standardization while simultaneously displaying a high degree of personally identifiable data. MethodsWe produced synthetic PRs (syPRs) out of real PRs (rePRs) using StyleGAN2-ADA by NVIDIA©. A survey was performed on 54 medical professionals and 33 dentistry students. They assessed 45 radiological images (20 rePRs, 20 syPRs, and 5 syPRcontrols) as real or synthetic and interpreted a single-image syPR according to the image quality (0–10) and 14 different items (agreement/disagreement). They also rated the importance for the profession (0–10). A follow-up was performed for test-retest reliability with >10 % of all participants. ResultsOverall, the sensitivity was 78.2 % and the specificity was 82.5 %. For professionals, the sensitivity was 79.9 % and the specificity was 82.3 %. For students, the sensitivity was 75.5 % and the specificity was 82.7 %. In the single syPR-interpretation image quality was rated at a median of 6 and 11 items were considered as agreement. The importance for the profession was rated at a median score of 7. The Test-retest reliability yielded a value of 0.23 (Cohen's kappa). ConclusionsThe study marks a comprehensive testing to demonstrate that GANs can produce synthetic radiological images that even health professionals can sometimes not differentiate from real radiological images, thereby being genuinely considered authentic. This enables their utilization and/or modification free from personally identifiable information. Clinical significanceSynthetic images can be used for university teaching and patient education without relying on patient-related data. They can also be utilized to upscale existing training datasets to improve the accuracy of AI-based diagnostic systems. The study thereby supports clinical teaching as well as diagnostic and therapeutic decision-making.

Deep multi-scale feature mixture model for image super-resolution with multiple-focal-length degradation

Single image super-resolution is a classic problem in computer vision. In recent years, deep learning-based models have achieved unprecedented success with this problem. However, most existing deep super-resolution models unavoidably produce degraded results when applied to real-world images captured by cameras with different focal lengths. The degradation in these images is called multiple-focal-length degradation, which is spatially variant and more complicated than the bicubic downsampling degradation. To address such a challenging issue, we propose a multi-scale feature mixture model in this paper. The proposed model can intensively exploit local patterns from different scales for image super-resolution. To improve the performance, we further propose a novel loss function based on the Laplacian pyramid, which guides the model to recover the information separately of different frequency subbands. Comprehensive experiments show that our proposed model has a better ability to preserve the structure of objects and generate high-quality images, leading to the best performance compared with other state-of-the-art deep single image super-resolution methods.