Single Image Research Articles

Joint Stroke Tracing and Correspondence for 2D Animation

To alleviate human labor in redrawing keyframes with ordered vector strokes for automatic inbetweening, we for the first time propose a joint stroke tracing and correspondence approach. Given consecutive raster keyframes along with a single vector image of the starting frame as a guidance, the approach generates vector drawings for the remaining keyframes while ensuring one-to-one stroke correspondence. Our framework trained on clean line drawings generalizes to rough sketches, and the generated results can be imported into inbetweening systems to produce inbetween sequences. Hence, the method is compatible with standard 2D animation workflow. An adaptive spatial transformation module (ASTM) is introduced to handle non-rigid motions and stroke distortion. We collect a dataset for training with 10k+ pairs of raster frames and their vector drawings with stroke correspondence. Comprehensive validations on real clean and rough animated frames manifest the effectiveness of our method and superiority to existing methods.

Dual-space high-frequency learning for transformer-based MRI super-resolution

Background and ObjectiveMagnetic resonance imaging (MRI) can provide rich and detailed high-contrast information of soft tissues, while the scanning of MRI is time-consuming. To accelerate MR imaging, a variety of Transformer-based single image super-resolution methods are proposed in recent years, achieving promising results thanks to their superior capability of capturing long-range dependencies. Nevertheless, most existing works prioritize the design of transformer attention blocks to capture global information. The local high-frequency details, which are pivotal to faithful MRI restoration, are unfortunately neglected. MethodsIn this work, we propose a high-frequency enhanced learning scheme to effectively improve the awareness of high frequency information in current Transformer-based MRI single image super-resolution methods. Specifically, we present two entirely plug-and-play modules designed to equip Transformer-based networks with the ability to recover high-frequency details from dual spaces: 1) in the feature space, we design a high-frequency block (Hi-Fe block) paralleled with Transformer-based attention layers to extract rich high-frequency features; while 2) in the image intensity space, we tailor a high-frequency amplification module (HFA) to further refine the high-frequency details. By fully exploiting the merits of the two modules, our framework can recover abundant and diverse high-frequency information, rendering faithful MRI super-resolved results with fine details. ResultsWe integrated our modules with six Transformer-based models and conducted experiments across three datasets. The results indicate that our plug-and-play modules can enhance the super-resolution performance of all foundational models to varying degrees, surpassing the capabilities of existing state-of-the-art single image super-resolution networks. ConclusionComprehensive comparison of super-resolution images and high-frequency maps from various methods, clearly demonstrating that our module possesses the capability to restore high-frequency information, showing huge potential in clinical practice for accelerated MRI reconstruction.

Improved Landsat Operational Land Imager (OLI) Cloud and Shadow Detection with the Learning Attention Network Algorithm (LANA)

Landsat cloud and cloud shadow detection has a long heritage based on the application of empirical spectral tests to single image pixels, including the Landsat product Fmask algorithm, which uses spectral tests applied to optical and thermal bands to detect clouds and uses the sun-sensor-cloud geometry to detect shadows. Since the Fmask was developed, convolutional neural network (CNN) algorithms, and in particular U-Net algorithms (a type of CNN with a U-shaped network structure), have been developed and are applied to pixels in square patches to take advantage of both spatial and spectral information. The purpose of this study was to develop and assess a new U-Net algorithm that classifies Landsat 8/9 Operational Land Imager (OLI) pixels with higher accuracy than the Fmask algorithm. The algorithm, termed the Learning Attention Network Algorithm (LANA), is a form of U-Net but with an additional attention mechanism (a type of network structure) that, unlike conventional U-Net, uses more spatial pixel information across each image patch. The LANA was trained using 16,861 512 × 512 30 m pixel annotated Landsat 8 OLI patches extracted from 27 images and 69 image subsets that are publicly available and have been used by others for cloud mask algorithm development and assessment. The annotated data were manually refined to improve the annotation and were supplemented with another four annotated images selected to include clear, completely cloudy, and developed land images. The LANA classifies image pixels as either clear, thin cloud, cloud, or cloud shadow. To evaluate the classification accuracy, five annotated Landsat 8 OLI images (composed of >205 million 30 m pixels) were classified, and the results compared with the Fmask and a publicly available U-Net model (U-Net Wieland). The LANA had a 78% overall classification accuracy considering cloud, thin cloud, cloud shadow, and clear classes. As the LANA, Fmask, and U-Net Wieland algorithms have different class legends, their classification results were harmonized to the same three common classes: cloud, cloud shadow, and clear. Considering these three classes, the LANA had the highest (89%) overall accuracy, followed by Fmask (86%), and then U-Net Wieland (85%). The LANA had the highest F1-scores for cloud (0.92), cloud shadow (0.57), and clear (0.89), and the other two algorithms had lower F1-scores, particularly for cloud (Fmask 0.90, U-Net Wieland 0.88) and cloud shadow (Fmask 0.45, U-Net Wieland 0.52). In addition, a time-series evaluation was undertaken to examine the prevalence of undetected clouds and cloud shadows (i.e., omission errors). The band-specific temporal smoothness index (TSIλ) was applied to a year of Landsat 8 OLI surface reflectance observations after discarding pixel observations labelled as cloud or cloud shadow. This was undertaken independently at each gridded pixel location in four 5000 × 5000 30 m pixel Landsat analysis-ready data (ARD) tiles. The TSIλ results broadly reflected the classification accuracy results and indicated that the LANA had the smallest cloud and cloud shadow omission errors, whereas the Fmask had the greatest cloud omission error and the second greatest cloud shadow omission error. Detailed visual examination, true color image examples and classification results are included and confirm these findings. The TSIλ results also highlight the need for algorithm developers to undertake product quality assessment in addition to accuracy assessment. The LANA model, training and evaluation data, and application codes are publicly available for other researchers.

Lightweight blueprint residual network for single image super-resolution

The application of deep convolutional neural networks (CNNs) makes the lightweight single image super-resolution (SISR) task develop rapidly in recent years. However, existing lightweight SISR networks for the terminals with limited computing resources are still not efficient enough. In order to further alleviate the model complexity while keeping remarkable reconstruction performance, in this paper, we propose a lightweight blueprint residual network (LBRN), which is a powerful CNN-based SR model while simultaneously maintaining extremely lightweight. Specifically, we design a novel lightweight blueprint residual block (LBRB) to learn the high-frequency information efficiently for lightweight SISR. The LBRB with fewer parameters consists of more efficient convolution operations to extract high-frequency information which helps recover more visual details. In addition, we propose an effective feature fusion residual block (EFFRB) which is composed of three LBRBs with convolutional kernels of different sizes. The EFFRB, which is designed with effective multi-scale receptive fields, can availably extract and fuse multi-scale high-frequency information to obtain discriminative feature, and it further improves model performance. Extensive experiments show that our LBRN achieves superior reconstruction performance quantitatively and visually compared with other lightweight state-of-the-art SR methods.

Performance of UAV-derived Normalized Difference Vegetation Index (NDVI) for Early Estimation of Rice Yield

Traditionally, the rice yield is measured after the harvest, which is too late then to revert any agronomic practices to improve yield. The aerial spectral reflectance images of crops captured by a multispectral camera mounted in an Unmanned Aerial Vehicle (UAV) are capable of quantitatively measuring the internal physiological condition of crops directly associated with their health status. The aim of this study was to evaluate the performance of UAV-multispectral image derived-NDVI as a method for estimating rice yield. Multispectral drone images including single band images were acquired from a controlled rice field at Rice Research and Development Institute (RRDI), Sri Lanka. Rice variety Bg 300 was cultivated in the Yala season under four levels of Nitrogen (N) fertilizer treatment plots. According to the regression co-relation analysis the derived NDVI values at 15 and 25 m flying heights from the rice crops at the booting stage were moderately (R2=0.65 and R2=0.67, p<0.05, respectively) associated with the actual yield. The derived NDVI values indicated the rice crop vigor at the booting stage is a useful indicator for early estimation of the rice yield prior to actual harvest.

Tumor conspicuity enhancement-based segmentation model for liver tumor segmentation and RECIST diameter measurement in non-contrast CT images

Background and objectiveLiver tumor segmentation (LiTS) accuracy on contrast-enhanced computed tomography (CECT) images is higher than that on non-contrast computed tomography (NCCT) images. However, CECT requires contrast medium and repeated scans to obtain multiphase enhanced CT images, which is time-consuming and cost-increasing. Therefore, despite the lower accuracy of LiTS on NCCT images, which still plays an irreplaceable role in some clinical settings, such as guided brachytherapy, ablation, or evaluation of patients with renal function damage. In this study, we intend to generate enhanced high-contrast pseudo-color CT (PCCT) images to improve the accuracy of LiTS and RECIST diameter measurement on NCCT images. MethodsTo generate high-contrast CT liver tumor region images, an intensity-based tumor conspicuity enhancement (ITCE) model was first developed. In the ITCE model, a pseudo color conversion function from an intensity distribution of the tumor was established, and it was applied in NCCT to generate enhanced PCCT images. Additionally, we design a tumor conspicuity enhancement-based liver tumor segmentation (TCELiTS) model, which was applied to improve the segmentation of liver tumors on NCCT images. The TCELiTS model consists of three components: an image enhancement module based on the ITCE model, a segmentation module based on a deep convolutional neural network, and an attention loss module based on restricted activation. Segmentation performance was analyzed using the Dice similarity coefficient (DSC), sensitivity, specificity, and RECIST diameter error. ResultsTo develop the deep learning model, 100 patients with histopathologically confirmed liver tumors (hepatocellular carcinoma, 64 patients; hepatic hemangioma, 36 patients) were randomly divided into a training set (75 patients) and an independent test set (25 patients). Compared with existing tumor automatic segmentation networks trained on CECT images (U-Net, nnU-Net, DeepLab-V3, Modified U-Net), the DSCs achieved on the enhanced PCCT images are both improved compared with those on NCCT images. We observe improvements of 0.696–0.713, 0.715 to 0.776, 0.748 to 0.788, and 0.733 to 0.799 in U-Net, nnU-Net, DeepLab-V3, and Modified U-Net, respectively, in terms of DSC values. In addition, an observer study including 5 doctors was conducted to compare the segmentation performance of enhanced PCCT images with that of NCCT images and showed that enhanced PCCT images are more advantageous for doctors to segment tumor regions. The results showed an accuracy improvement of approximately 3%–6%, but the time required to segment a single CT image was reduced by approximately 50 %. ConclusionsExperimental results show that the ITCE model can generate high-contrast enhanced PCCT images, especially in liver regions, and the TCELiTS model can improve LiTS accuracy in NCCT images.

Computer aided diagnosis of diabetic retinopathy based on multi-view joint learning

Diabetic retinopathy (DR) is a kind of ocular complication of diabetes, and its degree grade is an essential basis for early diagnosis of patients. Manual diagnosis is a long and expensive process with a specific risk of misdiagnosis. Computer-aided diagnosis can provide more accurate and practical treatment recommendations. In this paper, we propose a multi-view joint learning DR diagnostic model called RT2Net, which integrates the global features of fundus images and the local detailed features of vascular images to reduce the limitations of single fundus image learning. Firstly, the original image is preprocessed using operations such as contrast-limited adaptive histogram equalization, and the vascular structure of the extracted DR image is segmented. Then, the vascular image and fundus image are input into two branch networks of RT2Net for feature extraction, respectively, and the feature fusion module adaptively fuses the feature vectors’ output from the branch networks. Finally, the optimized classification model is used to identify the five categories of DR. This paper conducts extensive experiments on the public datasets EyePACS and APTOS 2019 to demonstrate the method’s effectiveness. The accuracy of RT2Net on the two datasets reaches 88.2% and 85.4%, and the area under the receiver operating characteristic curve (AUC) is 0.98 and 0.96, respectively. The excellent classification ability of RT2Net for DR can significantly help patients detect and treat lesions early and provide doctors with a more reliable diagnosis basis, which has significant clinical value for diagnosing DR.

Chromatically modeling the parsec-scale dusty structure in the center of NGC 1068

The Very Large Telescope Interferometer (VLTI) has been providing breakthrough images of the dust in the central parsecs of active galactic nuclei (AGNs), which is thought to be a key component of the AGN unification scheme and AGN host galaxy interaction. In single infrared bands, these images can enjoin multiple interpretations, some of which could challenge the unification scheme. This is the case for the archetypal type 2 AGN of NGC\,1068, whereby the degeneracy is reduced by multi-band temperature maps that are hindered by an ambiguity in the alignment between different single-band images. We aim to solve this problem by creating a chromatic model capable of simultaneously explaining the VLTI GRAVITY+MATISSE $2\ observations of AGNs hosted by NGC\,1068. We employed a simple disk and wind geometry populated by spherical black-body emitters and dust obscuration to create a versatile multi-wavelength modelling method for chromatic IR interferometric data of dusty objects. This simple geometry is capable of reproducing the spectro-interferometric data of NGC\,1068 from the K through N bands. It explains the complex single band images with obscuration and inclination effects, and it solves the alignment problem between bands. We find that the resulting model disk and wind geometry is consistent with previous studies of comparable and larger scales. For example, compared to molecular gas emission, our model wind position angle (PA) of $23^2_2 is close to the mas-scale outflowing CO(6--5) PA of $ seen with ALMA . The equivalent $90^ offset model disk PA is also consistent with the CO(6--5) disk axis of $112^ as well as the mas-scale disk axis from CO(2--1), CO(3--2), and HCO$^+$(4--3) of $115 circ$ . Furthermore, the resulting model images visually resemble the multiple achromatic image reconstructions of the same data when evaluated at the same wavelengths. We conclude that the IR emitting structure surrounding the AGN within NGC\,1068 can indeed be explained by the clumpy disk+wind iteration of the AGN unification scheme. Within the scheme, we find that it is best explained as a type 2 AGN and the obscuring dust chemistry can be explained by a mix of olivine silicates and $16 amorphous carbon.

Individual High-Rise Building Extraction from Single High-Resolution SAR Image Based on Part Model

Building extraction plays an important role in urban information analysis, which is helpful for urban planning and disaster monitoring. With the improvement of SAR resolution, rich detailed information in urban areas is revealed, but the discretized features also pose challenges for object detection. This paper addresses the problem of individual high-rise building extraction based on single high-resolution SAR image. Different from previous methods that require building facades to be presented in specific appearances, the proposed method is suitable for extraction of various types of high-rise buildings. After analyzing the SAR images of many types of high-rise buildings, we establish a unified high-rise building part model, on the basis of a scattering mechanism of building structures, to describe the facade characteristics of high-rise buildings, including a facade regularity part, facade bright line part, double bounce part, and their spatial topological relationships. A three-level high-rise building extraction framework, named HRBE-PM, is proposed based on the high-rise building part model. At the pixel level, a modified spot filter is used to extract bright spots and bright lines of different scales simultaneously to obtain salient features. At the structure level, building parts are generated based on the salient features according to semantic information. At the object level, spatial topological information between parts is introduced to generate building hypotheses. We define two indicators, completeness and compactness, to comprehensively evaluate each building hypothesis and select the optimal ones. After postprocessing, the final high-rise building extraction results are obtained. Experiments on two TerraSAR-X images show that the high-rise building extraction precision rate of the HRBE-PM method is above 85.29%, the recall rate is above 82.95%, and the F1-score is above 0.87. The results indicate that the HRBE-PM method can accurately extract individual high-rise buildings higher than 24 m in most dense scenes, regardless of building types.

Apple Fruit Edge Detection Model Using a Rough Set and Convolutional Neural Network.

Accurately and effectively detecting the growth position and contour size of apple fruits is crucial for achieving intelligent picking and yield predictions. Thus, an effective fruit edge detection algorithm is necessary. In this study, a fusion edge detection model (RED) based on a convolutional neural network and rough sets was proposed. The Faster-RCNN was used to segment multiple apple images into a single apple image for edge detection, greatly reducing the surrounding noise of the target. Moreover, the K-means clustering algorithm was used to segment the target of a single apple image for further noise reduction. Considering the influence of illumination, complex backgrounds and dense occlusions, rough set was applied to obtain the edge image of the target for the upper and lower approximation images, and the results were compared with those of relevant algorithms in this field. The experimental results showed that the RED model in this paper had high accuracy and robustness, and its detection accuracy and stability were significantly improved compared to those of traditional operators, especially under the influence of illumination and complex backgrounds. The RED model is expected to provide a promising basis for intelligent fruit picking and yield prediction.

Self-supervised monocular depth estimation on water scenes via specular reflection prior

Monocular depth estimation from a single image is an ill-posed problem for computer vision due to insufficient reliable cues as the prior knowledge. Besides the inter-frame supervision, namely stereo and adjacent frames, extensive prior information is available in the same frame. Reflections from specular surfaces, informative intra-frame priors, enable us to reformulate the ill-posed depth estimation task as a multi-view synthesis. This paper proposes the first self-supervision for deep-learning depth estimation on water scenes via intra-frame priors, known as reflection supervision and geometrical constraints. In the first stage, a water segmentation network is performed to separate the reflection components from the entire image. Next, we construct a self-supervised framework to predict the target appearance from reflections, perceived as other perspectives. The photometric re-projection error, incorporating SmoothL1 and a novel photometric adaptive SSIM, is formulated to optimize pose and depth estimation by aligning the transformed virtual depths and source ones. As a supplement, the water surface is determined from real and virtual camera positions, which complement the depth of the water area. Furthermore, to alleviate these laborious ground truth annotations, we introduce a large-scale water reflection scene (WRS) dataset rendered from Unreal Engine 4. Extensive experiments on the WRS dataset prove the feasibility of the proposed method compared to state-of-the-art depth estimation techniques.

CCNet: Collaborative Camouflaged Object Detection via decoder-induced information interaction and supervision refinement network

Recently, research based on the camouflaged object detection (COD) task has achieved great progress, while the collaborative camouflaged object detection (CoCOD) task is still lacking. Our research focuses on the simultaneous detection and localization of the collaborative camouflaged objects, i.e., CoCOD task. We use the cooperative information between a single image and a group of camouflage images to discover cooperative camouflaged objects effectively. In this paper, we propose a collaborative cross-scale feature learning network (CCNet). Our model is characterized by two innovative constructions: We proposed an edge augmentation module (EAM), which effectively extracts edge information of the camouflaged object and integrates it with the collaborative information employed in auxiliary supervision. In addition, we design a group decoder module (GDM) to extract and merge co-camouflage information. Extensive experiments on CoCOD8K datasets demonstrate that our CCNet significantly outperforms the existing 13 state-of-the-art COD and 6 state-of-the-art collaborative salient object detection (CoSOD) methods under six widely used evaluation metrics. The code will be available at: https://github.com/zc199823/CCNet--CoCOD.

Seeding detection and distribution evaluation using the developed automatic maize seeding machine

Seeding is a time-consuming and labour-intensive process in the detection of maize seeds germination rate. Automatic seeding can reduce labour requirements and improve efficiency. The quality of automatic seeding plays an important role in collecting accurate germination rates. In this research, a newly developed automatic maize seeding machine (AMSM) was used to detect the maize seeds and evaluate distribution by advanced computer vision and analysis methodologies. Three methods were utilized to detect maize seeds after they were loaded into the holes of the seed plate of AMSM: colour segmentation + connected component analysis, support vector machine (SVM) + sliding window, and YOLOV5. The results showed that compared with colour + connected component analysis and sliding window + SVM, YOLOV5 exhibited higher accuracy and improved computational efficiency. When the input image pixel size was 960 × 960, the precision of YOLOV5 + merge-NMS was 97.3 %, the recall was 0.918, the F1-score was 0.945, and the processing time of a single image was 0.38 ms, which is the optimal performance. YOLOV5 + merge-NMS demonstrated robust detection maize seeds in images with different illumination equalization processing. Based on maize seeds detection results, seeding distribution evaluation was realized with an average accuracy of 91.14 %. In addition, the normal seeding of the AMSM was 82.3 %. Utilization of YOLOV5 + merge-NMS enables rapid and precise detection of maize seeding while establishing a basis for accurately calculating seed distribution. The AMSM can meet the requirements of automatic maize seeds detection and distribution evaluation and the developed methodologies have the potential for automatic germination detection.

Deep learning-based frequency-multiplexing composite-fringe projection profilometry technique for one-shot 3D shape measurement

The pursuit of precise and efficient 3D shape measurement has long been a focal point within the fringe projection profilometry (FPP) domain. However, achieving precise 3D reconstruction for isolated objects from a single fringe image remains a challenging task in the field. In this paper, a deep learning-based frequency-multiplexing (FM) composite-fringe projection profilometry (DFCP) is proposed, where an end-to-end absolute phase retrieval network (APR-Net) is trained to directly recover the absolute phase map from a FM composite fringe pattern. The obtained absolute phase map exhibits exceptional precision and is devoid of spectrum crosstalk or leakage disturbance typically encountered in traditional FM techniques. APR-Net is intricately crafted, incorporating a nested strategy along with the concept of centralized information interaction. A diverse dataset encompassing various scenarios and free from spectrum aliasing is assembled to guide the training process. A seven-map loss calculation scheme is employed to guide the training process, of which the efficacy is proved through ablation experiments. In the first qualitative experiment, DFCP demonstrates comparable phase accuracy to ground truths with 47 fewer projected images, outperforming other three methods with mean absolute phase errors of 0.0052 rad, 0.1761 rad, 0.0169 rad, and 0.0139 rad. The second qualitative experiment and the quantitative evaluation respectively prove DFCP’s capability in high dynamic range 3D measurement and in precise 3D measurement, with sphere diameter errors of 0.0780 mm and 0.0726 mm, and a spherical centroid distance error of 0.0555 mm.

Пространственно-ориентационные параметры узнавания изображений лиц как предикторы симптомов печеночной недостаточности у подростков

Functional assessment of higher mental functions in case of intoxication or during treatment will make it possible to identify predictors of the symptoms of hepatic encephalopathy associated with portal hypertension. The study was aimed to determine the diagnostic predictors of the emergence of the earliest symptoms of hepatic encephalopathy in adolescents with portal hypertension. The study involved 60 adolescents aged 13–17 years: 28 males, 32 females. The experimental group included 30 adolescents with the diagnosis К76.6 Portal hypertension, unspecified form. The control group included 30 adolescents with normal somatic status, who had no mental disorders, traumatic brain injuries or severe infectious diseases of the brain (based on the records of the annual check-up). The studied groups were matched by sex, age, and social status. The proprietary method, Tobii EyeX hardware and software system (GazeControl software), and Cambridge Face Memory Test for Children (CFMT-C) were used. It has been found that recognition of single face images, multiple face images, and multiple face images camouflaged with noise by adolescents with portal hypertension is associated with the greater efforts (manifested in the increased number and duration of gaze fixations), than recognition of the above by adolescents with normal somatic status. The accuracy of recognition of single face images, multiple face images, and multiple face images camouflaged with noise shown by adolescents with portal hypertension experiencing the toxic effects associated with manifestations of hepatic encephalopathy is significantly reduced compared to that shown by adolescents with normal somatic status. The results obtained can be considered as the diagnostic predictors allowing one to trace the changes in the hepatic encephalopathy severity at various stages of treatment (including after surgical intervention).

GNDAN: Graph Navigated Dual Attention Network for Zero-Shot Learning.

Zero-shot learning (ZSL) tackles the unseen class recognition problem by transferring semantic knowledge from seen classes to unseen ones. Typically, to guarantee desirable knowledge transfer, a direct embedding is adopted for associating the visual and semantic domains in ZSL. However, most existing ZSL methods focus on learning the embedding from implicit global features or image regions to the semantic space. Thus, they fail to: 1) exploit the appearance relationship priors between various local regions in a single image, which corresponds to the semantic information and 2) learn cooperative global and local features jointly for discriminative feature representations. In this article, we propose the novel graph navigated dual attention network (GNDAN) for ZSL to address these drawbacks. GNDAN employs a region-guided attention network (RAN) and a region-guided graph attention network (RGAT) to jointly learn a discriminative local embedding and incorporate global context for exploiting explicit global embeddings under the guidance of a graph. Specifically, RAN uses soft spatial attention to discover discriminative regions for generating local embeddings. Meanwhile, RGAT employs an attribute-based attention to obtain attribute-based region features, where each attribute focuses on the most relevant image regions. Motivated by the graph neural network (GNN), which is beneficial for structural relationship representations, RGAT further leverages a graph attention network to exploit the relationships between the attribute-based region features for explicit global embedding representations. Based on the self-calibration mechanism, the joint visual embedding learned is matched with the semantic embedding to form the final prediction. Extensive experiments on three benchmark datasets demonstrate that the proposed GNDAN achieves superior performances to the state-of-the-art methods. Our code and trained models are available at https://github.com/shiming-chen/GNDAN.

The Utility of a Novel Stacked Microvascular Imaging for Enhanced Detection of Fibrosis in Chronic Liver Diseases

Ultrasonographic imaging plays a primary role to detect fibrotic changes in patients with chronic liver disease (CLD). To enhance detectability of fibrosis in its early stage, we developed a novel stacked microvascular imaging (SMVI) that enables continuous visualization of fibrotic changes in intrahepatic vessels. SMVI was produced by accumulating 3-5 seconds of high-definition color images in tilted-scan mode. An SMVI score was devised by quantitating three hallmark vascular changes in liver fibrosis in 0-2 grades (total 0-6): narrowing, caliber irregularity, and tortuosity. To evaluate the clinical utility of the SMVI score, 469 well-defined CLD patients were enrolled and subgrouped by the stage of liver fibrosis defined based on elastography: F0-1Low, F0-1High, F2, F3, and F4. The diagnostic performance of the SMVI score was compared to conventional B-mode liver morphology score and various laboratory test markers of fibrosis. Unlike conventional microvascular imaging that relies on a single image, SMVI enabled an undisrupted view of intrahepatic vessels for easy detection of fibrotic changes. SMVI detected microvascular narrowing in 92% at stage F0-1High. While detection rates for caliber irregularity and tortuosity were low at early stages but increased proportionately in advanced stages. Multiple logistic regression analysis revealed that SMVI score was most accurate in distinguishing F0-1Low from F0-1High cases compared to B-mode or laboratory test scores. SMVI provides enhanced vascular images of liver fibrosis in CLD, especially in its early stage. The SMVI score can be used as a primary tool for determining fibrotic stages in CLD.

BDR6D: Bidirectional Deep Residual Fusion Network for 6D Pose Estimation

Six-dimensional (6D) pose estimation is an important branch in the field of robotics focused on enhancing the ability of robots to manipulate and grasp objects. The latest research trend in 6D pose estimation is to directly predict the positions of two-dimensional (2D) keypoints from a single red, green, and blue (RGB) image through convolutional neural networks (CNNs) and establish a corresponding relationship with the three-dimensional (3D) keypoints of the model. Then, the perspective-n-point (PnP) algorithm is used to recover the 6D pose parameters. Currently, two challenges are encountered in pose estimation based on an RGB image. On the one hand, an RGB image lacks depth information, and it is thus difficult to directly obtain the corresponding geometric object information. On the other hand, when depth information is available, it is difficult to efficiently fuse the features of the RGB image with the features of the corresponding depth image. In this paper, we propose a bidirectional depth residual fusion network with a depth prediction (DP) network to estimate the 6D poses of objects (BDR6D). The BDR6D network predicts the depth information of objects using an RGB image, converts the depth information into point cloud information, and performs feature extraction and representation together with the RGB information during the feature extraction and representation stages. Specifically, the RGB image is fed into the BDR6D network, the DP network predicts the depth information of the objects in the image, and the depth map and RGB image are input into a point cloud network (PCN) and CNN, respectively, for feature extraction and representation. We build the bidirectional depth residual (BDR) structure so that the CNN and PCN can share information during feature extraction and representation. This approach allows the two networks to use each other’s local and global information to improve feature extraction and representation. For the keypoint selection stage, we propose an effective 2D keypoint selection method that considers the appearance and geometric information of the object of interest. We evaluate the proposed method with three benchmark datasets and compare it with other 6D pose estimation algorithms. The experimental results show that our method outperforms the state-of-the-art approach. Finally, we deploy our proposed method in conjunction with the Universal Robots 5 manipulator (UR5) robot to grasp and manipulate objects. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The purpose of this paper is to solve the problem of 6D pose estimation for robot grasping. The existing RGB image-based pose estimation approach faces two challenges. On the one hand, a single RGB image lacks depth information, so that it is difficult to directly obtain the corresponding geometric object information. On the other hand, when depth information is available, it is difficult to efficiently fuse the features of the RGB image with the features of the corresponding depth image. To solve the above problems, a novel network that can predict the depth information of objects from an RGB image and fuse the depth information with the RGB information to estimate the 6D pose of objects is proposed. Furthermore, we propose an effective 2D keypoint selection method that considers the appearance and geometric information of objects of interest. We evaluate the proposed approach based on three benchmark datasets and the UR5 robot platform and verify that our method is effective.

OIF-Net: An Optical Flow Registration-Based PET/MR Cross-Modal Interactive Fusion Network for Low-Count Brain PET Image Denoising.

The short frames of low-count positron emission tomography (PET) images generally cause high levels of statistical noise. Thus, improving the quality of low-count images by using image postprocessing algorithms to achieve better clinical diagnoses has attracted widespread attention in the medical imaging community. Most existing deep learning-based low-count PET image enhancement methods have achieved satisfying results, however, few of them focus on denoising low-count PET images with the magnetic resonance (MR) image modality as guidance. The prior context features contained in MR images can provide abundant and complementary information for single low-count PET image denoising, especially in ultralow-count (2.5%) cases. To this end, we propose a novel two-stream dual PET/MR cross-modal interactive fusion network with an optical flow pre-alignment module, namely, OIF-Net. Specifically, the learnable optical flow registration module enables the spatial manipulation of MR imaging inputs within the network without any extra training supervision. Registered MR images fundamentally solve the problem of feature misalignment in the multimodal fusion stage, which greatly benefits the subsequent denoising process. In addition, we design a spatial-channel feature enhancement module (SC-FEM) that considers the interactive impacts of multiple modalities and provides additional information flexibility in both the spatial and channel dimensions. Furthermore, instead of simply concatenating two extracted features from these two modalities as an intermediate fusion method, the proposed cross-modal feature fusion module (CM-FFM) adopts cross-attention at multiple feature levels and greatly improves the two modalities' feature fusion procedure. Extensive experimental assessments conducted on real clinical datasets, as well as an independent clinical testing dataset, demonstrate that the proposed OIF-Net outperforms the state-of-the-art methods.

Advancing multimodal medical image fusion: an adaptive image decomposition approach based on multilevel Guided filtering.

With the rapid development of medical imaging methods, multimodal medical image fusion techniques have caught the interest of researchers. The aim is to preserve information from diverse sensors using various models to generate a single informative image. The main challenge is to derive a trade-off between the spatial and spectral qualities of the resulting fused image and the computing efficiency. This article proposes a fast and reliable method for medical image fusion depending on multilevel Guided edge-preserving filtering (MLGEPF) decomposition rule. First, each multimodal medical image was divided into three sublayer categories using an MLGEPF decomposition scheme: small-scale component, large-scale component and background component. Secondly, two fusion strategies-pulse-coupled neural network based on the structure tensor and maximum based-are applied to combine the three types of layers, based on the layers' various properties. The three different types of fused sublayers are combined to create the fused image at the end. A total of 40 pairs of brain images from four separate categories of medical conditions were tested in experiments. The pair of images includes various case studies including magnetic resonance imaging (MRI) , TITc, single-photon emission computed tomography (SPECT) and positron emission tomography (PET). We included qualitative analysis to demonstrate that the visual contrast between the structure and the surrounding tissue is increased in our proposed method. To further enhance the visual comparison, we asked a group of observers to compare our method's outputs with other methods and score them. Overall, our proposed fusion scheme increased the visual contrast and received positive subjective review. Moreover, objective assessment indicators for each category of medical conditions are also included. Our method achieves a high evaluation outcome on feature mutual information (FMI), the sum of correlation of differences (SCD), Qabf and Qy indexes. This implies that our fusion algorithm has better performance in information preservation and efficient structural and visual transferring.