Holistically-nested Edge Detection Research Articles

Building Edge Detection Technology From Remote Sensing Image Based On NSCT And Tensor Voting

An edge detection technology based on the combination of non-downsampling contour wave transform (NSCT) and tensor voting is proposed, which aims to obtain more accurate and detailed edge information of buildings in remote sensing images. Firstly, NSCT is used for image decomposition to obtain the subband frequency information of different scales and angles. Then, position encoding is performed on these subband coefficients to obtain second-order symmetric tensors at the corresponding positions. Tensors of different scales and angles at the same position are weighted and summed to complete feature fusion. Finally, the edge features of the image are obtained based on tensor voting theory. The experimental results show that compared to common edge detection technologies, such as Canny, Fast Edge (Fast edge detection using structured forests) and HED (Holistically-Nested Edge Detection), our method can more accurately and intensively reflect the boundaries of buildings and the edge information of roofs, providing better support for the analysis of building types and architectural styles. Compared to the HED method, which is based on deep learning, our method improves PSNR and SSIM metrics by 0.98 and 0.03, respectively.

Internet of Things Based Oil Pipeline Spill Detection System Using Deep Learning and LAB Colour Algorithm

Given the role that pipelines play in transporting crude oil, which is considered the basis of the global economy and across different environments, hundreds of studies revolve around providing the necessary protection for it. Various technologies have been employed in this pursuit, differing in terms of cost, reliability, and efficiency, among other factors. Computer vision has emerged as a prominent technique in this field, albeit requiring a robust image-processing algorithm for spill detection. This study employs image segmentation techniques to enable the computer to interpret visual information and images effectively. The research focuses on detecting spills in oil pipes caused by leakage, utilizing images captured by a drone equipped with a Raspberry Pi and Pi camera. These images, along with their global positioning system (GPS) location, are transmitted to the base station using the message queuing telemetry transport Internet of Things (MQTT IoT) protocol. At the base station, deep learning techniques, specifically Holistically-Nested Edge Detection (HED) and extreme inception (Xception) networks, are employed for image processing to identify contours. The proposed algorithm can detect multiple contours in the images. To pinpoint a contour with a black color, representative of an oil spill, the CIELAB color space (LAB) algorithm effectively removes shadow effects. If a contour is detected, its area and perimeter are calculated to determine whether it exceeds a certain threshold. The effectiveness of the proposed system was tested on Iraqi oil pipeline systems, demonstrating its capability to detect spills of different sizes.

PLANIMETRIC RAIL POSITIONING USING UAV PHOTOGRAMMETRY: TOWARDS AUTOMATED AND SAFE RAILWAY INFRASTRUCTURE MONITORING

Abstract. The maintenance of railway infrastructure requires detailed inspection of track assets including the rails, sleepers, fasteners, and tie plates. Current methods using total stations and measurement trains are costly and the subsequent data processing is often manual. A potent alternative is the use of Unmanned Aerial Vehicles (UAV) to investigate track asset deviations. However, the potential degree of automation and the overall accuracy is still very much the subject of ongoing research. In this work, a potential pipeline is investigated for the planimetric inspection of rails using UAV photogrammetry. Specifically, state-of-the-art line detectors such as Holistically-Nested Edge Detection, DexiNed, Segment Anything Model, and Mobile Line Segment Detection are combined with logical filtering to assess the localisation and gauge of the rails. The experiments indicate that the accuracy and detection rate are promising. Overall, the proposed method is a promising step towards affordable and safe railway infrastructure inspection.

Deep Semantic Segmentation New Model of Natural and Medical Images

Semantic segmentation is the most significant deep learning technology. At present, automatic assisted driving (Autopilot) is widely used in real-time driving, but if there is a deviation in object detection in real vehicles, it can easily lead to misjudgment. Turning and even crashing can be quite dangerous. This paper seeks to propose a model for this problem to increase the accuracy of discrimination and improve security. It proposes a Convolutional Neural Network (CNN)+ Holistically-Nested Edge Detection (HED) combined with Spatial Pyramid Pooling (SPP). Traditionally, CNN is used to detect the shape of objects, and the edge may be ignored. Therefore, adding HED increases the robustness of the edge, and finally adds SPP to obtain modules of different sizes, and strengthen the detection of undetected objects. The research results are trained in the CityScapes street view data set. The accuracy of Class mIoU for small objects reaches 77.51%, and Category mIoU for large objects reaches 89.95%.

Holistically Nested Edge Detection and particle filtering for subtle vibration extraction

Extracting and analyzing subtle motion via optical sensing techniques is becoming a more important research area within the vibration community. Computer vision approaches and machine learning architecture have been adopted to contest the complexity concerning hand instrumentation of sensors. Pre-treatment using a stochastic pattern for optical analysis or traditional wired sensing is not always permitted for large structures. In this work, Holistically Nested Edge Detection (HED) is utilized to highlight high-spatial-frequency content features in images for structural dynamic analysis. Following the accentuation of edge features, phase-based motion magnification (PMM) is used to amplify motion that is not visible to the naked eye. This permits the band-passing of higher order resonant frequencies for structural dynamic evaluation. Implementation of two-dimensional particle filtering (PF) is utilized to track the accentuated edge features in the presence of ghosting artifacts that appear post magnification. Within this work, compared to traditional edge-thresholding, the combination of HED and PF (HED-PF) is more proficient to handle noise and ultimately improves magnified image quality and modal parameter extraction. Uncertainties related to using PF for vibration measurement are analytically presented to provide guidance on the behavior of the algorithm. To demonstrate the effectiveness of HED-PF, experimental modal testing takes place on both a tier-structure and cantilever beam to demonstrate the approaches’ capability in handling varying sorts of excitation. Finally, introduction of background occlusions and non-ideal lighting scenarios are used to test the limitations of the approach. Ultimately, this will highlight HED-PF’s robust tracking capabilities compared to the current state of the art.

UNDERGROUND CRUDE OIL PIPELINE LEAKAGE DETECTION USING DEXINED DEEP LEARNING TECHNIQUES AND LAB COLOR SPACE

Computer vision plays a big role in pipeline leakage detection systems and is one of the latest techniques. Still, it requires a powerful image-processing algorithm to detect objects. The purpose of this work is to develop and implement spill detection in oil pipes caused by leakage using images taken by a drone equipped with a Raspberry Pi 4. The acquired images are sent to the base station along with the global positioning system (GPS) location of the captured images via the message queuing telemetry transport Internet of Things (MQTT IoT) protocol. At the base station, images are processed to identify contours by dense extreme inception networks for edge detection(DexiNed) deep learning techniques based on holistically-nested edge detection(HED) and extreme inception (Xception) networks. This algorithm is capable of finding many contours in images. To find a contour with black color, the CIELAB color space (LAB) has been used. The proposed algorithm removes small contours and computes the area of the remaining contours. If the contour is above the threshold value, it is considered a spill; otherwise, it will be saved in a database for further inspection. For testing purposes, three different spill areas were implemented with spill sizes of (1 m^2,2 m^2 ,and 3 m^2). Images have been captured at three different heights (5 m, 10 m, and 15 m) by the drone used to capture the images. The result shows that effective detection has been obtained at 10 meters high. To monitor the entire system, a web application has been integrated into the base station.

The Style Transfer of Photos and Landscape Paintings Based on CycleGAN Combined with Nested Edge Detection Module

Chinese landscape paintings, especially Chinese landscape ink paintings have always been the treasure of the Chinese, or even the world culture. Due to the advancement of the science and knowledge, there are always interests for finding ways to produce Chinese landscape paintings through the way of the technology. In this paper, based on the original CycleGAN model, we are trying to transfer a real-world photograph to a typical Chinese landscape painting. To be specific, the brushwork in Chinese ink paintings contains enormous numbers of distinct and technical brush strokes which makes it extremely hard for CycleGAN to recognize and transfer Chinese ink painting. We improved the brushstroke effect of Chinese ink painting still by incorporating the Holistically Nested Edge Detection (HED) method into the CycleGAN model. The HED module puts the source image and the corresponding generated image into the convolution layer of five stages to generate their respective edge maps. The balanced cross-entropy loss calculated with edge maps is added to the total loss of CycleGAN to train the generator which is confronted with discriminator to improve output results. The addition of HED enables the extraction of edge features from the images, preserving the structural information and enhancing the accuracy of the style transfer, so the detailed of the Chinese ink painting can be transferred better in this sense. Experimental process with model building, training, validating, and predicting concludes that, complementing HED method into the original CycleGAN models well preserved distinct features in the conversion processes from real-world photographs to traditional Chinese landscape paintings.

Suppressing the mechanochromism of flexible photonic crystals.

Photonic crystal slabs (PCS) are a promising platform for optical biosensing. Yet, flexible applications based on PCS for biosensing have been limited, as the mechanical properties influence the optical ones. Here, we show the suppression of the mechanochromism effect for flexible PCS. We obtained flexible photonic crystal slabs by sputtering of a dielectric 100 nm Nb2O5 high refractive index layer onto a flexible nanostructured polydimethylsiloxane (PDMS) substrate with 370 nm grating period. The PCS exhibit a guided mode resonance at around 650 nm. We demonstrate that these flexible photonic crystal slabs show less than 0.5 nm resonance shift for 4% strain and call them stabilized PCS (sPCS). We compare this to a resonance shift of ∼21 nm for ∼4% strain of a flexible photonic crystal with a flexible nanoparticle high index layer (mechanochromatic PCS, mPCS). This high resonance shift is expected from the Bragg equations, where 4% grating period change correspond to approximately 4% change of the resonance wavelength (i.e., ∼26 nm at a resonance wavelength of 650 nm), if changes in the mode effective refractive index are neglected. In a stretch series we obtain color-to-strain dependencies of 4.79 nm/% strain for mPCS and 0.11 nm/% strain for our stabilized sPCS. We analyze the suppression of the mechanochromism with detailed microscopy results. We observe that fissures and fractures form in the rigid waveguiding layer of the sPCS upon mechanical stress. An algorithm based on Holistically-Nested Edge Detection (HED) is used for automated counting of cracks. Rigid photonic crystal cells with sizes on the order of 10 µm to 100 µm are formed that explain the stable optical properties. Even more stable optical properties with less than 0.03 nm wavelength shift per 1% strain are demonstrated for sPCS with an additional dielectric 100 nm SiO2 low index layer beneath the Nb2O5 waveguide layer decoupling the waveguide further from the flexible PDMS substrate.

Seismic interpolation via multi-scale HU-Net

The recovery of the seismic data with missing traces is essential for seismic interpretation and reservoir characterization. The deep learning (DL) based models have shown the promising results for the challenging task of seismic data interpolation. However, interpolating the large gap missing traces is a big challenge for seismic interpolation. The holistically-nested edge detection (HED) model can learn the multi-scale and multi-level features for the holistic image training and prediction. The U-Net model has a large number of feature channels in the upsampling part, which allow it propagate the feature maps to the higher resolution layers. Utilizing the advantages of these two state-of-art DL models, we propose a novel convolution neural network (CNN) based model for seismic interpolation, named the HU-Net model, by integrating the holistically-nested module and the U-net model. To formalize this, we add five activation layers followed by the U-Net model to replace the origin sigmoid layer. We then use five side branches to connect the decoder with these activation layers and output multi-scale features. Thus, our proposed HU-Net cannot only recognize the objects in big images, but also reconstruct the multi-scale features. We investigate the ability of our proposed HU-Net for interpolating the missing seismic traces with big gaps and reconstruct the fining sequence texture simultaneously. The synthetic and field data examples illustrate the effectiveness and robustness of our proposed HU-Net model. The detailed comparisons also demonstrate the superiority of our proposed model over the conventional methods.

Algorithm for Intelligent Recognition Low-Grade Seismic Faults Using Codec Target Edges

The development of remaining oil plays an important role in increasing the late production, and low-grade faults seriously impacts residual oil exploration and development. Low-grade faults have a small fault displacement, brief extension, and strong concealment, which hinders their prediction using the typical semantic segmentation network. To intelligently identify low-grade faults, we designed a codec target edge detection technique. For the network to fully learn the low-grade faults information, we constructed the encoder using dilated convolution. Next, we introduced an attention mechanism to the decoder to improve capture of location information from a shallow network and semantic information from a deep network. Finally, the multi-scale fusion decoder outputs fault information of different scales, which further improves the identification accuracy of low-grade faults. The training model is applied to simulated data and actual seismic data through ablation experiments. The results show that this method can effectively identify low-grade faults and overcomes problems such as blurred fault cross-location, thicker edge contour lines, lower detection accuracy, and less training data. Compared with conventional a Holistically-Nested Edge Detection (HED) and a semantic segmentation (UNet), fault misidentification is reduced, fault continuity is increased, and fault accuracy is improved. providing technical support for the exploration and development of remaining oil and increasing the recovery rate of old oil fields.

Fingertip Detection Algorithm Based on Maximum Discrimination HOG Feature in Complex Background

Gesture fingertip detection plays a vital role in human-computer interaction applications such as VR and robot control. To improve the accuracy of fingertip detection in complex background, this paper proposes a fingertip detection algorithm based on a novel maximum discrimination HOG feature. Firstly, the Holistically-nested edge detection algorithm is used to detect the edge of hand images and non-hand images, and the HOG features of the contour edge are extracted to obtain the training set of positive and negative samples to reduce the influence of illumination, color, and texture on fingertip detection. Secondly, the maximum discrimination features are filtered from the positive sample set by a custom maximum discrimination feature filter and stored as a feature dictionary. The filtered maximum discrimination features and negative sample set are input into the XGBoost classifier for training and the voting rights classifier is obtained. Filtering maximum discrimination features can reduce the interference of irrelevant features and improve the performance of the classifier. Then, the KNN algorithm is used to find the best match in the dictionary, and the final fingertip position is obtained by the Meanshift algorithm. Finally, the algorithm in this paper is tested. The test results show that the accuracy of fingertip detection in this paper can reach 99 %, and the detected RMSE is kept in the range of 5 pixels, which is higher than that based on YCbCr skin color segmentation, YOLO target detection, and YOLO-YCbCr.

A Deep Learning Based Method to Delineate the Wet/Dry Shoreline and Compute Its Elevation Using High-Resolution UAS Imagery

Automatically detecting the wet/dry shoreline from remote sensing imagery has many benefits for beach management in coastal areas by enabling managers to take measures to protect wildlife during high water events. This paper proposes the use of a modified HED (Holistically-Nested Edge Detection) architecture to create a model for automatic feature identification of the wet/dry shoreline and to compute its elevation from the associated DSM (Digital Surface Model). The model is generalizable to several beaches in Texas and Florida. The data from the multiple beaches was collected using UAS (Uncrewed Aircraft Systems). UAS allow for the collection of high-resolution imagery and the creation of the DSMs that are essential for computing the elevations of the wet/dry shorelines. Another advantage of using UAS is the flexibility to choose locations and metocean conditions, allowing to collect a varied dataset necessary to calibrate a general model. To evaluate the performance and the generalization of the AI model, we trained the model on data from eight flights over four locations, tested it on the data from a ninth flight, and repeated it for all possible combinations. The AP and F1-Scores obtained show the success of the model’s prediction for the majority of cases, but the limitations of a pure computer vision assessment are discussed in the context of this coastal application. The method was also assessed more directly, where the average elevations of the labeled and AI predicted wet/dry shorelines were compared. The absolute differences between the two elevations were, on average, 2.1 cm, while the absolute difference of the elevations’ standard deviations for each wet/dry shoreline was 2.2 cm. The proposed method results in a generalizable model able to delineate the wet/dry shoreline in beach imagery for multiple flights at several locations in Texas and Florida and for a range of metocean conditions.

Ghost edge detection based on HED network

In this paper, we present an edge detection scheme based on ghost imaging (GI) with a holistically-nested neural network. The so-called holistically-nested edge detection (HED) network is adopted to combine the fully convolutional neural network (CNN) with deep supervision to learn image edges effectively. Simulated data are used to train the HED network, and the unknown object’s edge information is reconstructed from the experimental data. The experiment results show that, when the compression ratio (CR) is 12.5%, this scheme can obtain a high-quality edge information with a sub-Nyquist sampling ratio and has a better performance than those using speckle-shifting GI (SSGI), compressed ghost edge imaging (CGEI) and subpixel-shifted GI (SPSGI). Indeed, the proposed scheme can have a good signal-to-noise ratio performance even if the sub-Nyquist sampling ratio is greater than 5.45%. Since the HED network is trained by numerical simulations before the experiment, this proposed method provides a promising way for achieving edge detection with small measurement times and low time cost.Graphical

Application of Deep Learning Workflow for Autonomous Grain Size Analysis.

Traditional grain size determination in materials characterization involves microscopy images and a laborious process requiring significant manual input and human expertise. In recent years, the development of computer vision (CV) has provided an alternative approach to microstructural characterization with preliminary implementations greatly simplifying the grain size determination process. Here, an end-to-end workflow to measure grain size in microscopy images without any manual input is presented. Following the ASTM standards for grain size determination, results from the line intercept (Heyn’s method) and planimetric (Saltykov’s method) approaches are used as the baseline. A pre-trained holistically nested edge detection (HED) model is used for CV-based edge detection, and the results are further compared to the classic Canny edge detection method. Post-processing was performed using open-source image processing packages to extract the grain size. In optical microscope images, the pre-trained HED model achieves much higher accuracy than the Canny edge detection method while reducing the image processing time by one to two orders of magnitude compared to traditional methods. The effects of morphological operations on the predicted grain size accuracy are also explored. Overall, the proposed end-to-end convolutional neural network (CNN)-based workflow can significantly reduce the processing time while maintaining the same accuracy as the traditional manual method.

An extraction method of fish vector field for marine cage culture based on UAV vision

To obtain the motion state of a school of fish in marine cage culture and improve the automatic detection capability for large areas of marine culture, this paper proposes a fish-motion vector-field extraction method for marine cage culture, which primarily comprises cage region extraction and motion vector calculation. The proposed method uses an unmanned aerial vehicle (UAV) as the carrier, and the interference of marine environment is fully considered. First, blurred images are discarded, and the culture area is obtained by a holistically-nested edge detection (HED) network. Second, the validity of the current frame is determined by the local binary patterns (LBPs) of the feature image. Third, the images are transformed, and the vector-field information of fish swarm is obtained using the particle image velocimetry (PIV) analysis method. This study integrates traditional image analysis and deep network technology while also fully considering the image distortion caused by UAV flight attitude changes in the actual detection process. Finally, images of Takifugu rubripe net cages were used to verify the method. The results show that the proposed method has strong practicability and applicability.

Video frame prediction with dual-stream deep network emphasizing motions and content details

Video frame prediction is both challenging and critical for computer vision. Though the research on predicting video frames has gradually shifted from pixel-law based methods to motion based ones, existing predictors often generate ambiguous future frames, especially for long-term predictions. This paper proposes a composed model to generate future frames with more details. First, to further exploit motion information, we design a single motion decoder to strengthen the efficiency of the motion encoder in the original motion-content network (MCnet). Second, to alleviate prediction ambiguousness, we use both edges with and without semantic meanings from the holistically-nested edge detection (HED) module as content details. Third, based on the conclusion that the mean squared error (MSE) loss and the traditional generative adversarial learning framework cause the unsatisfied predictions of MCnet, we design a composite loss function that can guide our model to simultaneously focus on motions and content details. Also, based on the abovementioned conclusion, we finally embed our model in an improved generative adversarial network, which further enhances its performance. Experimental results on the benchmark KTH and UCF101 datasets show that our model outperforms the state-of-the-art predictors, such as the basic MCnet, the predictive neural network (PredNet), and the PredNet with a reduced-gate convolutional network (rgc-PredNet), in terms of peak signal to noise ratio (PSNR) and structural similarity index measure (SSIM), especially for long-term video frame prediction.

Approach to edge detection in medical imaging through histogram analysis and morphological gradient

Edge detection takes importance in image processing systems for computer-aided diagnosis, wheresharp changes in pixel intensity are analyzed to obtain fast and accurate information about regions ofinterest to the specialist. A method for feature enhancement and edge detection in medical imageswas developed using image processing by analyzing the pixel distribution histogram andmorphological gradient operation. Images from the MINI MIAS dataset and the COVID-CT datasetwere used. The method is based on image processing and is applied to mammography and chest CTimages, where blur filtering steps are accompanied by morphological gradient filtering, in addition toobtaining the threshold for edge detection by analyzing the point of maximum pixel concentrationaccording to the distribution histogram. The processing is presented in a graphical user interfacedeveloped in Python language. The method is validated by comparison with other edge detectiontechniques such as the Canny Algorithm, and with deep learning methods such as Holistically-NestedEdge Detection. The proposed method improves image quality in both mammograms and CT scanscompared to other techniques. It also presents the best performance considering internal and externaledge detection, as well as an average response time of 1.054 seconds and 2.63 % of Central Processing Unit requirement. The developed system is presented as a support tool for use in computer-aideddiagnosis processes due to its high efficiency in edge detection.

Glass Defect Detection via Multi-Scale Feature Fusion

Glass defect detection is significant in glass industry. However, most of the existing methods for glass defect detection currently still rely on manual screening with high-cost and poor-efficiency. To address this issue, we propose a glass defect detection method using multi-scale feature fusion strategy. Specifically, we first propose an algorithm based on pix2pix to realize the edge extraction. Then, we propose a glass defect detector based on both local and global features. Comprehensive experiments are conducted on collected glass dataset from factories. The experimental results demonstrate that our method outperforms conventional methods including the traditional image processing and Holistically-Nested Edge Detection (HED), with the precision rate (PR) up to 97%, the false precision rate (FP) below 2% and the total accuracy rate (ACC) of glass defect detection up to 98%.

An Image Edge Detection Algorithm Based on Multi-Feature Fusion

Edge detection is one of the core steps of image processing and computer vision. Accurate and fine image edge will make further target detection and semantic segmentation more effective. Holistically-Nested edge detection (HED) edge detection network has been proved to be a deep-learning network with better performance for edge detection. However, it is found that when the HED network is used in overlapping complex multi-edge scenarios for automatic object identification. There will be detected edge incomplete, not smooth and other problems. To solve these problems, an image edge detection algorithm based on improved HED and feature fusion is proposed. On the one hand, features are extracted using the improved HED network: the HED convolution layer is improved. The residual variable convolution block is used to replace the normal convolution enhancement model to extract features from edges of different sizes and shapes. Meanwhile, the empty convolution is used to replace the original pooling layer to expand the receptive field and retain more global information to obtain comprehensive feature information. On the other hand, edges are extracted using Otsu algorithm: Otsu-Canny algorithm is used to adaptively adjust the threshold value in the global scene to achieve the edge detection under the optimal threshold value. Finally, the edge extracted by improved HED network and Otsu-Canny algorithm is fused to obtain the final edge. Experimental results show that on the Berkeley University Data Set (BSDS500) the optimal data set size (ODS) F-measure of the proposed algorithm is 0.793; the average precision (AP) of the algorithm is 0.849; detection speed can reach more than 25 frames per second (FPS), which confirms the effectiveness of the proposed method.

Boundary Enhancement Semantic Segmentation for Building Extraction From Remote Sensed Image

Image processing via convolutional neural network (CNN) has been developed rapidly for remote sensing technology. Moreover, techniques for accurately extracting building footprints from remote sensed images have attracted considerable interest owing to their wide variety of common applications, including monitoring natural disasters and urban development. Extraction of building footprints can be performed easily by semantic segmentation using U-Net-like CNN architectures. However, obtaining precise boundaries of segmentation masks remains challenging due to various impediments surrounding target objects. In this study, we propose a method to elaborate edges of buildings detected in remote sensed images to enhance the boundaries of segmentation masks. The proposed method adopts <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">holistically nested edge detection (HED)</i> , which extracts edge features at an encoder of a given architecture. In the proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">boundary enhancement (BE) module</i> , an extracted edge and segmentation mask are combined, sharing mutual information. To enable the proposed method efficiently to adapt to a wide variety of conditions, we design a distinctive approach adopting a HED unit and BE module, which is applicable to various semantic segmentation networks containing encoder-decoder structures. Experiments were conducted on five different datasets (DeepGlobe, Urban3D, WHU [high-resolution (HR), low-resolution (LR)], and Massachusetts). The results demonstrate that our proposed approaches improved on the performance of prior methods for extracting building footprints. Comparative experiments were conducted on various backbone architectures including U-Net, ResUNet++, TernausNet, and U-shape spatial pyramid pooling (USPP) to ensure the effectiveness of the proposed method. Based on various evaluation metrics and qualitative analysis, our results show that the proposed method achieved improved performance compared with prior methods for all datasets and backbone networks.