Refinement Network Research Articles

Human body contour extraction method based on human skeleton key point guidance

By extracting human contours from 2D images captured by the camera and then obtaining human size data, the cost of garment custom measurement can be effectively reduced and the efficiency of custom measurement can be improved. The extraction of human contours plays an important role in the collection of online human size data. We propose a method to extract human contours by fusing the prior information of human skeleton key points into the salience target detection network. Specifically, the skeleton key point information extracted based on OpenPose is fused into the encoder-decoder network for rough detection of the human body target, and the residual refinement network is used to fine-adjust the human body matting, so as to achieve accurate human contour extraction. In this paper, the accuracy and superiority of the algorithm are verified in the public data set P3M-10K of human body matting and applied to the 2D body measurement WeChat applet on mobile phone and computer website.

An adaptive feature refinement network for pixel-level segmentation of surface defect

Abstract In the field of industrial production, automatic surface defect segmentation plays a crucial role in improving product quality and production efficiency. However, due to the interference of complex backgrounds and uneven illumination, efficient and accurate segmentation of surface defect remains a significant challenge. To address these issues, this work proposes an adaptive feature refinement U-shaped network (U-Net) for pixel-level segmentation of surface defect. The proposed framework draws inspiration from the classic U-Net architecture, and the pre-trained EfficientNet-B0 is selected as the encoder part to extract the multi-scale context information. Then, a novel adaptive feature refinement module, which further learns the global channel-wise dependencies and captures cross-channel local spatial patterns from multi-level features, is introduced to enhance the robustness of feature representation. Finally, these enhanced encoder features are up-sampled, and are further fed into the decoder part to obtain pixel-level defect segmentation results. To assess the validity and accuracy of the proposed model, this network is validated through extensive experiments on four different types of defect datasets. Experimental results show that the proposed model performs well on these datasets, with the maximum mean intersection over union (MIoU) of 91.63%. Meanwhile, the model is compared with some U-Net-based models and other state-of-the-art segmentation models, and it can also achieve competitive performance on MIoU scores. With the acceleration of the graphics processing unit, the proposed network can achieve a detection speed of 82 frames per second, which can meet the needs of real-time detection in the industrial fields.

Edge-aware interactive refinement network for strip steel surface defects detection

Edge-aware interactive refinement network for strip steel surface defects detection

Neural Disparity Refinement.

We propose a framework that combines traditional, hand-crafted algorithms and recent advances in deep learning to obtain high-quality, high-resolution disparity maps from stereo images. By casting the refinement process as a continuous feature sampling strategy, our neural disparity refinement network can estimate an enhanced disparity map at any output resolution. Our solution can process any disparity map produced by classical stereo algorithms, as well as those predicted by modern stereo networks or even different depth-from-images approaches, such as the COLMAP structure-from-motion pipeline. Nonetheless, when deployed in the former configuration, our framework performs at its best in terms of zero-shot generalization from synthetic to real images. Moreover, its continuous formulation allows for easily handling the unbalanced stereo setup very diffused in mobile phones.

CRNet: Cascaded Refinement Network for polyp segmentation

Technology for automatic segmentation plays a crucial role in the early diagnosis and treatment of ColoRectal Cancer (CRC). Existing polyp segmentation methods often focus on advanced feature extraction while neglecting detailed low-level features, This somewhat limits the enhancement of segmentation performance. This paper proposes a new technique called the Cascaded Refinement Network (CRNet), designed to improve polyp segmentation performance by combining low-level and high-level features through a cascaded contextual network structure. To accurately capture the morphological variations of polyps and enhance the clarity of segmentation boundaries, we have designed the Multi-Scale Feature Optimization (MFO) module and the Contextual Edge Guidance (CEG) module. Additionally, to further enhance feature fusion and utilization, we introduced the Cascaded Local Feature Fusion (CLFF) module, which effectively integrates cross-layer correlations, allowing the network to understand complex polyp structures better. By conducting a large number of experiments, our model achieved a 0.3% and 3.1% higher mDice score than the latest MMFIL-Net in the two main datasets of Kvasir-SEG and CVC-ClinicDB, respectively. Ablation studies show that MFO improves the baseline score by 4%, and the network without CLFF and CEG results in a reduction of 2.4% and 1.7% in mDice scores, respectively. This further validates the contribution of each module to the polyp segmentation performance. CRNet enhances model performance through the introduction of multiple modules but also increases model complexity. Future work will explore how to reduce computational complexity and improve inference speed while maintaining high performance. The source code for this paper can be found at https://github.com/l1986036/CRNet.

Category-Level Pose Estimation and Iterative Refinement for Monocular RGB-D Image

Category-level pose estimation is proposed to predict the 6D pose of objects under a specific category and has wide applications in fields such as robotics, virtual reality, and autonomous driving. With the development of VR/AR technology, pose estimation has gradually become a research hotspot in 3D scene understanding. However, most methods fail to fully utilize geometric and color information to solve intra-class shape variations, which leads to inaccurate prediction results. To solve the above problems, we propose a novel pose estimation and iterative refinement network, use an attention mechanism to fuse multi-modal information to obtain color features after a coordinate transformation, and design iterative modules to ensure the accuracy of object geometric features. Specifically, we use an encoder-decoder architecture to implicitly generate a coarse-grained initial pose and refine it through an iterative refinement module. In addition, due to the differences between rotation and position estimation, we design a multi-head pose decoder that utilizes the local geometry and global features. Finally, we design a transformer-based coordinate transformation attention module to extract pose-sensitive features from RGB images and supervise color information by correlating point cloud features in different coordinate systems. We train and test our network on the synthetic dataset CAMERA25 and the real dataset REAL275. Experimental results show that our method achieves state-of-the-art performance on multiple evaluation metrics.

Semi-supervised spatial-temporal calibration and semantic refinement network for video polyp segmentation

Automated video polyp segmentation (VPS) was of vitality for the early prevention and diagnosis of colorectal cancer (CRC). However, existing deep learning-based automatic polyp segmentation methods mainly focused on independent static images and struggled to perform well due to neglecting spatial–temporal relationships among successive video frames, while requiring massive frame-by-frame annotations. To better alleviate these challenges, we proposed a novel semi-supervised spatial–temporal calibration and semantic refinement network (STCSR-Net) dedicated to VPS, which simultaneously considered both the inter-frame temporal consistency in video clips and intra-frame semantic-spatial information. It was composed of a segmentation pathway and a propagation pathway by use of a co-training scheme for supervising the predictions on un-annotated images in a semi-supervised learning fashion. Specifically, we proposed an adaptive sequence calibration (ASC) block in segmentation pathway and a dynamic transmission calibration (DTC) block in propagation pathway to fully take advantage of valuable temporal cues and maintain the prediction temporally consistent among consecutive frames. Meanwhile, in these two branches, we introduced residual block (RB) to suppress irrelevant noisy information and highlight rich local boundary details of polyp lesions, while constructed multi-scale context extraction (MCE) module to enhance multi-scale high-level semantic feature expression. On that basis, we designed progressive adaptive context fusion (PACF) module to gradually aggregate multi-level features under the guidance of reinforced high-level semantic information for eliminating semantic gaps among them and promoting the discrimination capacity of features for targeting polyp objects. Through synergistic combination of RB, MCE and PACF modules, semantic-spatial correlations on polyp lesions within each frame could be established. Coupled with the context-free loss, our model merged feature representations of neighboring frames to diminish the dependency on varying contexts within consecutive frames and strengthen its robustness. Extensive experiments substantiated that our model with 100% annotation ratio achieved state-of-the-art performance on challenging datasets. Even trained under 50% annotation ratio, our model exceled significantly existing state-of-the-art image-based and video-based polyp segmentation models on the newly-built local TRPolyp dataset by at least 1.3% and 0.9% enhancements in both mDice and mIoU, whilst exhibited comparable performance to top rivals attained through using fully supervised approach on publicly available CVC-612, CVC-300 and ASU-Mayo-Clinic benchmarks. Notably, our model showcased exceptionally well in videos containing complex scenarios like motion blur and occlusion. Beyond that, it also harvested approximately 0.794 mDice and 0.707 mIoU at an inference of 0.036 s per frame in endoscopist-machine competition, which outperformed junior and senior endoscopists as well as almost matched with those of expert ones. The strong capability of the proposed STCSR-Net held promise in improving quality of VPS, accentuating model’s adaptability and potential in real-world clinical scenarios.

AER-Net: Attention-Enhanced Residual Refinement Network for Nuclei Segmentation and Classification in Histology Images.

The acurate segmentation and classification of nuclei in histological images are crucial for the diagnosis and treatment of colorectal cancer. However, the aggregation of nuclei and intra-class variability in histology images present significant challenges for nuclei segmentation and classification. In addition, the imbalance of various nuclei classes exacerbates the difficulty of nuclei classification and segmentation using deep learning models. To address these challenges, we present a novel attention-enhanced residual refinement network (AER-Net), which consists of one encoder and three decoder branches that have same network structure. In addition to the nuclei instance segmentation branch and nuclei classification branch, one branch is used to predict the vertical and horizontal distance from each pixel to its nuclear center, which is combined with output by the segmentation branch to improve the final segmentation results. The AER-Net utilizes an attention-enhanced encoder module to focus on more valuable features. To further refine predictions and achieve more accurate results, an attention-enhancing residual refinement module is employed at the end of each encoder branch. Moreover, the coarse predictions and refined predictions are combined by using a loss function that employs cross-entropy loss and generalized dice loss to efficiently tackle the challenge of class imbalance among nuclei in histology images. Compared with other state-of-the-art methods on two colorectal cancer datasets and a pan-cancer dataset, AER-Net demonstrates outstanding performance, validating its effectiveness in nuclear segmentation and classification.

A Shadow Imaging Bilinear Model and Three-Branch Residual Network for Shadow Removal.

The current shadow removal pipeline relies on the detected shadow masks, which have limitations for penumbras and tiny shadows, and results in an excessively long pipeline. To address these issues, we propose a shadow imaging bilinear model and design a novel three-branch residual (TBR) network for shadow removal. Our bilinear model reveals the single-image shadow removal process and can explain why simply increasing the brightness of shadow areas cannot remove shadows without artifacts. We considerably shorten the shadow removal pipeline by modeling illumination compensation and developing a single-stage shadow removal network without additional detection and refinement networks. Specifically, our network consists of three task branches, i.e., shadow image reconstruction, shadow matte estimation, and shadow removal. To merge these three branches and enhance the shadow removal branch, we design a model-based TBR module. Multiple TBR modules are cascaded to generate an intensive information flow and facilitate feature integration among the three branches. Thus, our network ensures the fidelity of nonshadow areas and restores the light intensity of shadow areas through three-branch collaboration. Extensive experiments demonstrate that our method outperforms the state-of-the-art methods. The model and code are available at https://github.com/nachifur/TBRNet.

Learning Feature Embedding Refiner for Solving Vehicle Routing Problems.

While the encoder-decoder structure is widely used in the recent neural construction methods for learning to solve vehicle routing problems (VRPs), they are less effective in searching solutions due to deterministic feature embeddings and deterministic probability distributions. In this article, we propose the feature embedding refiner (FER) with a novel and generic encoder-refiner-decoder structure to boost the existing encoder-decoder structured deep models. It is model-agnostic that the encoder and the decoder can be from any pretrained neural construction method. Regarding the introduced refiner network, we design its architecture by combining the standard gated recurrent units (GRU) cell with two new layers, i.e., an accumulated graph attention (AGA) layer and a gated nonlinear (GNL) layer. The former extracts dynamic graph topological information of historical solutions stored in a diversified solution pool to generate aggregated pool embeddings that are further improved by the GRU, and the latter adaptively refines the feature embeddings from the encoder with the guidance of the improved pool embeddings. To this end, our FER allows current neural construction methods to not only iteratively refine the feature embeddings for boarder search range but also dynamically update the probability distributions for more diverse search. We apply FER to two prevailing neural construction methods including attention model (AM) and policy optimization with multiple optima (POMO) to solve the traveling salesman problem (TSP) and the capacitated VRP (CVRP). Experimental results show that our method achieves lower gaps and better generalization than the original ones and also exhibits competitive performance to the state-of-the-art neural improvement methods.

Does the FARNet neural network algorithm accurately identify Posteroanterior cephalometric landmarks?

BackgroundWe explored whether the feature aggregation and refinement network (FARNet) algorithm accurately identified posteroanterior (PA) cephalometric landmarks.MethodsWe identified 47 landmarks on 1,431 PA cephalograms of which 1,177 were used for training, 117 for validation, and 137 for testing. A FARNet-based artificial intelligence (AI) algorithm automatically detected the landmarks. Model effectiveness was calculated by deriving the mean radial error (MRE) and the successful detection rates (SDRs) within 2, 2.5, 3, and 4 mm. The Mann-Whitney U test was performed on the Euclidean differences between repeated manual identifications and AI trials. The direction in differences was analyzed, and whether differences moved in the same or opposite directions relative to ground truth on both the x and y-axis.ResultsThe AI system (web-based CranioCatch annotation software (Eskişehir, Turkey)) identified 47 anatomical landmarks in PA cephalograms. The right gonion SDRs were the highest, thus 96.4, 97.8, 100, and 100% within 2, 2.5, 3, and 4 mm, respectively. The right gonion MRE was 0.94 ± 0.53 mm. The right condylon SDRs were the lowest, thus 32.8, 45.3, 54.0, and 67.9% within the same thresholds. The right condylon MRE was 3.31 ± 2.25 mm. The AI model’s reliability and accuracy were similar to a human expert’s. AI was better at four skeleton points than the expert, whereas the expert was better at one skeletal and seven dental points (P < 0.05). Most of the points exhibited significant deviations along the y-axis. Compared to ground truth, most of the points in AI and the second trial showed opposite movement on the x-axis and the same on the y-axis.ConclusionsThe FARNet algorithm streamlined orthodontic diagnosis.

Effective Bi-decoding networks for rail-surface defect detection by knowledge distillation

No-service rail-surface defect detection is a crucial method for assessing the quality of railroad tracks. However, the low-contrast and dark-tone characteristics of track-surface textures pose challenges to current defect-monitoring techniques. Real-time and on-site online inspections are important to ensure safe railway operation; however, most complex models for no-service inspections are difficult to deploy on mobile devices. To address these challenges and overcome the detection difficulties associated with complex scenes, we designed a knowledge distillation-based double decoding-layer refinement network (EBDNet-KD). The first decoding process is guided by a bimodal high-level semantic feature map obtained by extending the attention-based graph convolution to incrementally enhance the dual-stream features and obtain an image restoration prior. A divide-and-conquer decoder is then designed to distinguish features using different decoding layers. The prior is then used in the second decoding layer, which enables the bimodal features to interact fully and obtain the final prediction map. We introduce a knowledge distillation strategy that enables a lightweight, compact student network to learn a complex teacher network’s feature extraction process. This facilitates pixel-consistent learning of the knowledge within the bi-decoder layer, as well as bidirectional learning of the focused contextual response knowledge to optimize the model. The EBDNet-KD significantly reduces computational costs while guaranteeing performance with a parameter count of only 28 M. EBDNet-KD demonstrated superior performance over 15 state-of-the-art methods in experiments conducted on NEU RSDDS-AUG, an industrial RGB-depth dataset. We assessed the generalizability of EBDNet-KD by evaluating its performance on three additional public datasets, yielding competitive results. The source code and results can be found at https://github.com/Wuyue15/EBDNet.

Irregular adaptive refinement network for semantic segmentation of high-resolution remote sensing images

The semantic segmentation of high-resolution remote sensing images has broad application prospects in land cover classification, road extraction, urban planning and other fields. To alleviate the influence of the large data volume and complex background of high-resolution remote sensing images, the usual approach is to downsample them or cut them into small pieces for separate processing. Even if combining the two methods can improve the segmentation efficiency, it ignores the differences between the middle and the edge regions. Therefore, we consider the characteristics of large and irregular region in high-resolution remote sensing images, and then propose an irregular adaptive refinement network to locate the irregular edge region, which will be refined adaptively. Specifically, on the basis of effectively preserving the global and local information, the prediction confidence is calculated to locate pixel points that are poorly segmented, so as to form irregular regions requiring further refinement, avoiding to ‘over-refine’ intermediate region with good segmentation. At the same time, considering the difference in the refinement degree of different pixels, we propose to adaptively integrate the local segmentation results to refine the coarse segmentation results. In addition, in order to bridge the gap between the two extreme ends of the scale space, we introduce a multi-scale framework. Finally, we conducted experiments on the Deepglobe dataset showing that the proposed method performed 0.37% to 0.87% better than the previous state-of-the-art methods in terms of mean Intersection over Union (mIoU).

Feature-aware and iterative refinement network for camouflaged object detection

Feature-aware and iterative refinement network for camouflaged object detection

Enhanced Cross Layer Refinement Network for robust lane detection across diverse lighting and road conditions

With the rapid development of autonomous driving technology, lane detection, a key component of intelligent vehicle systems, is crucial for ensuring road safety and efficient vehicle navigation. In this paper, a new lane detection method is proposed to address the problem of degraded performance of existing lane detection methods when dealing with complex road environments. The proposed method evolves from the original Cross Layer Refinement Network (CLRNet) by incorporating two of our carefully designed core components: the Global Feature Optimizer (GFO) and the Adaptive Lane Geometry Aggregator (ALGA). The GFO is a multi-scale attention mechanism that mimics the human visual focusing ability, effectively filtering out unimportant information and focusing on the image regions most relevant to the task. The ALGA is a shape feature-aware aggregation module that utilizes the shape prior of lanes to enhance the correlation of anchor points in an image, better fusing global and local information. By integrating both components into CLRNet, an enhanced version called Enhanced CLRNet (E-CLRNet) is presented, which exhibits higher performance stability in complex roadway scenarios. Experiments on the CULane dataset reveal that E-CLRNet demonstrates superior performance stability over the original CLRNet in complex scenarios, including curves, shadows, missing lines, and dazzling light conditions. In particular, in the curves, the F1 score of E-CLRNet is improved by almost 3% over the original CLRNet. This study not only improves the accuracy and performance stability of lane detection but also provides a new solution for the application of autonomous driving technology in complex environments, which promotes the development of intelligent vehicle systems.

Enhanced Feature Refinement Network Based on Depthwise Separable Convolution for Lightweight Image Super-Resolution

Image super-resolution (SR) techniques aim to enhance the clarity and realism of images. Recently, a wide range of excellent SR algorithms with powerful characterization capabilities have emerged and are widely used. However, there are still challenges and room for improvement in designing a lighter and more edge-friendly SR networks for hardware devices. In this paper, we propose a lightweight enhanced feature refinement network (EFRN) based on depthwise separatable convolution for SR reconstruction. The core network components consist of multiple enhanced feature refinement blocks (EFRB), which fully fuse channel features to extract more accurate low-frequency information based on the attention of different channels. In addition, a lightweight residual block (LRB) and a lightweight dual attention block (LDAB) are designed to enhance network information extraction with minimal parameter cost. We improve the feature refinement by using 1 × 1 convolution instead of a channel selection operation to reduce the dimensionality of the features and extract the refined features more efficiently. Finally, to achieve better reconstruction performance, the depth and number of channels of the network are expanded while keeping the total number of parameters at a low level. Extensive experiments have been conducted to demonstrate the superiority of our EFRN over other mainstream SR algorithms in terms of reconstruction results and the number of parameters.

DenseFusion-DA2: End-to-End Pose-Estimation Network Based on RGB-D Sensors and Multi-Channel Attention Mechanisms.

Notably, 6D pose estimation is a critical technology that enables robotics to perceive and interact with their operational environment. However, occlusion causes a loss of local features, which, in turn, restricts the estimation accuracy. To address these challenges, this paper proposes an end-to-end pose-estimation network based on a multi-channel attention mechanism, DA2Net. Firstly, a multi-channel attention mechanism, designated as "DA2Net", was devised using A2-Nets as its foundation. This mechanism is constructed in two steps. In the first step, the essential characteristics are extracted from the global feature space through the second-order attention pool. In the second step, a feature map is generated by the integration of position and channel attention. Subsequently, the extracted key features are assigned to each position of the feature map, enhancing both the feature representation capacity and the overall performance. Secondly, the designed attention mechanism is introduced into both the feature fusion and pose iterative refinement networks to enhance the network's capacity to acquire local features thus improving its overall performance. The experimental results demonstrated that the estimation accuracy of DenseFusion-DA2 on the LineMOD dataset was approximately 3.4% higher than that of DenseFusion. Furthermore, the estimation accuracy surpassed that of PoseCNN, PVNet, SSD6D, and PointFusion by 8.3%, 11.1%, 20.3%, and 23.8%, respectively. The estimation accuracy also shows a significant advantage on the Occluded LineMOD and HR-Vision datasets. This research not only presents a more efficient solution for robot perception but also introduces novel ideas and methods for technological advancements and applications in related fields.

Redesigned generative framework with refinement network for improved localization of image forgeries

Redesigned generative framework with refinement network for improved localization of image forgeries

DeepFaceReshaping: Interactive deep face reshaping via landmark manipulation

Deep generative models allow the synthesis of realistic human faces from freehand sketches or semantic maps. However, although they are flexible, sketches and semantic maps provide too much freedom for manipulation, and thus, are not easy for novice users to control. In this study, we present DeepFaceReshaping, a novel landmark-based deep generative framework for interactive face reshaping. To edit the shape of a face realistically by manipulating a small number of face landmarks, we employ neural shape deformation to reshape individual face components. Furthermore, we propose a novel Transformer-based partial refinement network to synthesize the reshaped face components conditioned on the edited landmarks, and fuse the components to generate the entire face using a local-to-global approach. In this manner, we limit possible reshaping effects within a feasible component-based face space. Thus, our interface is intuitive even for novice users, as confirmed by a user study. Our experiments demonstrate that our method outperforms traditional warping-based approaches and recent deep generative techniques.

Label refinement network from synthetic error augmentation for medical image segmentation

Deep convolutional neural networks for image segmentation do not learn the label structure explicitly and may produce segmentations with an incorrect structure, e.g., with disconnected cylindrical structures in the segmentation of tree-like structures such as airways or blood vessels. In this paper, we propose a novel label refinement method to correct such errors from an initial segmentation, implicitly incorporating information about label structure. This method features two novel parts: (1) a model that generates synthetic structural errors, and (2) a label appearance simulation network that produces segmentations with synthetic errors that are similar in appearance to the real initial segmentations. Using these segmentations with synthetic errors and the original images, the label refinement network is trained to correct errors and improve the initial segmentations. The proposed method is validated on two segmentation tasks: airway segmentation from chest computed tomography (CT) scans and brain vessel segmentation from 3D CT angiography (CTA) images of the brain. In both applications, our method significantly outperformed a standard 3D U-Net, four previous label refinement methods, and a U-Net trained with a loss tailored for tubular structures. Improvements are even larger when additional unlabeled data is used for model training. In an ablation study, we demonstrate the value of the different components of the proposed method.