Patch-based Approach Research Articles

Efficient and Adaptable Patch-Based Crack Detection

Road crack inspection is an important process to maintain the quality of roads for safety issues. The manual road inspection is laborious and time consuming. Thus, an automatic road crack detection is essential to make the inspection process easier and faster. Normally, this crack detection is conducted in real-time with low power computational devices and limited memory. Consequently, many former approaches adopted the patch-based approach to reduce computation for single forward, where the input image is divided into several non-overlapping patches. The generated patches are processed by independent CNN models to capture the crack position. Yet, this approach can lead to disintegrating issues because each patch is processed independently when the CNN fails to detect some patches of the crack. In this study, the improved patch-based crack detector is proposed, and the global patch analyzer is adopted to handle the above issue by considering the relation of each patch processing. Moreover, the proposed model also involves more features such as multiple decoder design and automatic resource mapper to yield superior results than that of the state-of-the-art methods in terms of the speed and accuracy as examined by extensive experiments.

Learning local depth regression from defocus blur by soft-assignment encoding.

We present a novel, to the best of our knowledge, patch-based approach for depth regression from defocus blur. Most state-of-the-art methods for depth from defocus (DFD) use a patch classification approach among a set of potential defocus blurs related to a depth, which induces errors due to the continuous variation of the depth. Here, we propose to adapt a simple classification model using a soft-assignment encoding of the true depth into a membership probability vector during training and a regression scale to predict intermediate depth values. Our method uses no blur model or scene model; it only requires a training dataset of image patches (either raw, gray scale, or RGB) and their corresponding depth label. We show that our method outperforms both classification and direct regression on simulated images from structured or natural texture datasets, and on raw real data having optical aberrations from an active DFD experiment.

Deep Neural Network Concept for a Blind Enhancement of Document-Images in the Presence of Multiple Distortions

In this paper, we propose a new convolutional neural network (CNN) architecture for improving document-image quality through decreasing the impact of distortions (i.e., blur, shadows, contrast issues, and noise) contained therein. Indeed, for many document-image processing systems such as OCR (optical character recognition) and document-image classification, the real-world image distortions can significantly degrade the performance of such systems in a way such that they become merely unusable. Therefore, a robust document-image enhancement model is required to preprocess the involved document images. The preprocessor system developed in this paper places “deblurring” and “noise removal and contrast enhancement” in two separate and sequential submodules. In the architecture of those two submodules, three new parts are introduced: (a) the patch-based approach, (b) preprocessing layer involving Gabor and Blur filters, and (c) the approach using residual blocks. Using these last-listed innovations results in a very promising performance when compared to the related works. Indeed, it is demonstrated that even extremely strongly degraded document images that were not previously recognizable by an OCR system can now become well-recognized with a 91.51% character recognition accuracy after the image enhancement preprocessing through our new CNN model.

649 CNN-based histopathology image analysis for early-stage melanoma recurrence

Emerging data suggests that the majority of melanoma mortality occurs in patients with recurrence of disease that was initially early-stage (Stage I or II). Thus, there is a need for prognostic tools to identify patients at high risk of recurrence. This study examines the capability of weakly-supervised Convolutional Neural Networks (CNNs) for detecting prognostic signals of early-stage melanoma recurrence using whole-slide histopathology images (WSIs). We collected 224 WSIs from 58 patients with recurrence and 203 WSIs from 58 patients without recurrence and a minimum of 5-year follow-up. Each gigapixel WSI was processed into 512 x 512 pixel patches. We annotated regions of tumor for 50 patients and performed a two-step analysis. First, a model was trained to classify melanoma and non-melanoma patches. Second, a model was trained to differentiate between patches at high- and low-risk of recurrence. We compared performance of two CNN architectures, VGG16 and ResNet50, using 5-fold cross-validation. The median study follow-up for non-recurrent tumors was 6.5 years. Both architectures successfully identified melanoma patches: VGG16 (ACC 94.5%, AUC 98.9%) and ResNet50 (ACC 95%, AUC 99%). For recurrence prediction, VGG16 achieved slightly better patient-level performance (ACC 85.3%, AUC 93.5%) compared to ResNet50 (ACC 85.3%, AUC 91.3%). We demonstrate that a state-of-the-art CNN pipeline can accurately detect prognostic signals for early-stage melanoma recurrence using only WSIs. Our patch-based approach achieved these results without requiring detailed region of interest annotation. This approach can be used to automatically identify tumor areas with high-risk features, enabling subsequent molecular analyses of these regions to understand specific mechanisms for recurrence.

Efficient pavement distress classification via deep patch soft selective learning and knowledge distillation

Pavement distress classification is a vital step for automatic pavement inspection and maintenance. Recently, patch-based approaches have achieved promising performances and thus extensive attention in this field. However, these methods simply assume that all patches contribute equally to the distress classification, leading to weakly discriminating abilities of models. Moreover, their tedious processes also lead to low efficiency in inference. The authors present a novel patch-based pavement distress classification approach named Deep Patch Soft Selective Learning, which addresses these issues. Similar to other patch-based approaches, Deep Patch Soft Selective Learning partitions the pavement images into patches and aggregates the patch features to accomplish the task. To address the first issue, a succinct Soft Patch Feature Selection Network is introduced to assess the importance of each patch to the distress classification with a score based on its feature. These scores will be considered as patch-wise weights for feature aggregation. In such a manner, the most discriminative patches are selected in a soft way, and thereby benefit the final classification. To address the inference efficiency issue, knowledge distillation is leveraged to transfer the classification knowledge from Deep Patch Soft Selective Learning to the image-based approaches, such as EfficientNet-B3. This distilled model enables incorporating both the advantages of patch-based approaches in classification performance and the advantages of image-based approaches in inference efficiency. Extensive experiments on a large-scale pavement image dataset named CQU-BPDD demonstrate that Deep Patch Soft Selective Learning outperforms the baselines by a large margin of +7.4% in P@R on the detection task and +3.1% in F1 on the recognition task while enjoying 2.1x throughput of baselines.

Patch Diffusion: A General Module for Face Manipulation Detection

Detection of manipulated face images has attracted a lot of interest recently. Various schemes have been proposed to tackle this challenging problem, where the patch-based approaches are shown to be promising. However, the existing patch-based approaches tend to treat different patches equally, which do not fully exploit the patch discrepancy for effective feature learning. In this paper, we propose a Patch Diffusion (PD) module which can be integrated into the existing face manipulation detection networks to boost the performance. The PD consists of Discrepancy Patch Feature Learning (DPFL) and Attention-Aware Message Passing (AMP). The DPFL effectively learns the patch features by a newly designed Pairwise Patch Loss (PPLoss), which takes both the patch importance and correlations into consideration. The AMP diffuses the patches through attention-aware message passing in a graph network, where the attentions are explicitly computed based on the patch features learnt in DPFL. We integrate our PD module into four recent face manipulation detection networks, and carry out the experiments on four popular datasets. The results demonstrate that our PD module is able to boost the performance of the existing networks for face manipulation detection.

Deep learning for necrosis detection using canine perivascular wall tumour whole slide images

Necrosis seen in histopathology Whole Slide Images is a major criterion that contributes towards scoring tumour grade which then determines treatment options. However conventional manual assessment suffers from inter-operator reproducibility impacting grading precision. To address this, automatic necrosis detection using AI may be used to assess necrosis for final scoring that contributes towards the final clinical grade. Using deep learning AI, we describe a novel approach for automating necrosis detection in Whole Slide Images, tested on a canine Soft Tissue Sarcoma (cSTS) data set consisting of canine Perivascular Wall Tumours (cPWTs). A patch-based deep learning approach was developed where different variations of training a DenseNet-161 Convolutional Neural Network architecture were investigated as well as a stacking ensemble. An optimised DenseNet-161 with post-processing produced a hold-out test F1-score of 0.708 demonstrating state-of-the-art performance. This represents a novel first-time automated necrosis detection method in the cSTS domain as well specifically in detecting necrosis in cPWTs demonstrating a significant step forward in reproducible and reliable necrosis assessment for improving the precision of tumour grading.

Deep neural network trained on gigapixel images improves lymph node metastasis detection in clinical settings

The pathological identification of lymph node (LN) metastasis is demanding and tedious. Although convolutional neural networks (CNNs) possess considerable potential in improving the process, the ultrahigh-resolution of whole slide images hinders the development of a clinically applicable solution. We design an artificial-intelligence-assisted LN assessment workflow to facilitate the routine counting of metastatic LNs. Unlike previous patch-based approaches, our proposed method trains CNNs by using 5-gigapixel images, obviating the need for lesion-level annotations. Trained on 5907 LN images, our algorithm identifies metastatic LNs in gastric cancer with a slide-level area under the receiver operating characteristic curve (AUC) of 0.9936. Clinical experiments reveal that the workflow significantly improves the sensitivity of micrometastasis identification (81.94% to 95.83%, P < .001) and isolated tumor cells (67.95% to 96.15%, P < .001) in a significantly shorter review time (−31.5%, P < .001). Cross-site evaluation indicates that the algorithm is highly robust (AUC = 0.9829).

A Hybrid Deep Learning and Visualization Framework for Pushing Behavior Detection in Pedestrian Dynamics.

Crowded event entrances could threaten the comfort and safety of pedestrians, especially when some pedestrians push others or use gaps in crowds to gain faster access to an event. Studying and understanding pushing dynamics leads to designing and building more comfortable and safe entrances. Researchers—to understand pushing dynamics—observe and analyze recorded videos to manually identify when and where pushing behavior occurs. Despite the accuracy of the manual method, it can still be time-consuming, tedious, and hard to identify pushing behavior in some scenarios. In this article, we propose a hybrid deep learning and visualization framework that aims to assist researchers in automatically identifying pushing behavior in videos. The proposed framework comprises two main components: (i) Deep optical flow and wheel visualization; to generate motion information maps. (ii) A combination of an EfficientNet-B0-based classifier and a false reduction algorithm for detecting pushing behavior at the video patch level. In addition to the framework, we present a new patch-based approach to enlarge the data and alleviate the class imbalance problem in small-scale pushing behavior datasets. Experimental results (using real-world ground truth of pushing behavior videos) demonstrate that the proposed framework achieves an 86% accuracy rate. Moreover, the EfficientNet-B0-based classifier outperforms baseline CNN-based classifiers in terms of accuracy.

Application of optimal clustering and metric learning to patch-based anomaly detection

Anomaly detection in computer vision is a process of identifying observations or events deviated from the norm. Pixel- and patch-based approaches for locating abnormal regions suffer from a tremendously expensive computational complexity due to repeated calculations inherently required. The approach introduced in this letter aims to reducing such computational burden by employing an optimal clustering method for features extracted from a deep network-based encoder, and a metric learning which provides a way to bias the clusters found by the clustering algorithm. A set of loss functions is defined and used for training the encoder. After the training steps, the clustering result obtained from the training dataset is saved and used in the inference phase by measuring distances between the features of the input data and the clusters. The proposed approach has been applied to a patch-based anomaly detection, Patch-SVDD, to clarify the effectiveness of the idea. The experiment carried out with the implementation on MVTec-AD dataset results in improved detection speed by 10∼35% and better detection accuracy for many image classes, as well. In addition, the ablation study conducted proves the validity of the ideas introduced in the proposed approach.

AlzVNet: A volumetric convolutional neural network for multiclass classification of Alzheimer’s disease through multiple neuroimaging computational approaches

Alzheimer's disease is a degenerative neurological disease that causes a loss of cognitive skills and has no known treatment. Alzheimer's disease (AD) must be detected early, before symptoms appear, in order to be treated effectively. In this study, we used a deep learning approach called a convolutional neural network to classify Alzheimer's disease into three categories using a neuroimaging biomarker called T1w-MRI. Our research is the first to look at the results of three neuroimaging computational approaches in a systematic way (3D subject-level, 3D patch-based and slice-based). To show Alzheimer detection using deep convolutional neural networks, three distinct Slice Based methods are used (Subset selection method, uniform selection method, Interpolation zoom method). For 3D patch-based approaches, we investigated the classification accuracy of distinct non-overlapping patches ranging in size from small to medium to large (from 32, 40, 48, 56, 64, 72, 80, till 88). Our findings revealed that 1) our 3-class classification model performed best, with 98.3 percent accuracy percent (highest accuracy obtained until now as per our best knowledge); 2) The 3D Subject-level approach was the most efficient, followed by 3D-patch-based and then Slice-based approaches, with classification accuracy of 98.26 percent, 97.48 percent, and 95.40 percent, respectively; and 3) The same network had the most accuracy for bigger patches (size 72, 80, 88), followed by medium-sized (size 56, 64) to tiny patches (size 32, 40, 48). Large patches had a classification accuracy of 97.48 percent, while medium patches had a classification accuracy of 96.62 percent, and small patches had an accuracy of 86.49 percent. 4)) Even slice selection and interpolation selection exceeded subset slice selection with three-class classification accuracy of 95.37 percent and 94.57 percent, respectively, compared to 92.57 percent for subset slice selection.

Deepfake Detection of Occluded Images Using a Patch-Based Approach

Deepfake Detection of Occluded Images Using a Patch-Based Approach

A Patch-Based Low-Rank Minimization Approach for Speckle Noise Reduction in Ultrasound Images

A Patch-Based Low-Rank Minimization Approach for Speckle Noise Reduction in Ultrasound Images

BrainSec: Automated Brain Tissue Segmentation Pipeline for Scalable Neuropathological Analysis.

As neurodegenerative disease pathological hallmarks have been reported in both grey matter (GM) and white matter (WM) with different density distributions, automating the segmentation process of GM/WM would be extremely advantageous for aiding in neuropathologic deep phenotyping. Standard segmentation methods typically involve manual annotations, where a trained researcher traces the delineation of GM/WM in ultra-high-resolution Whole Slide Images (WSIs). This method can be time-consuming and subjective, preventing a scalable analysis on pathology images. This paper proposes an automated segmentation pipeline (BrainSec) combining a Convolutional Neural Network (CNN) module for segmenting GM/WM regions and a post-processing module to remove artifacts/residues of tissues. The final output generates XML annotations that can be visualized via Aperio ImageScope. First, we investigate two baseline models for medical image segmentation: FCN, and U-Net. Then we propose a patch-based approach, BrainSec, to classify the GM/WM/background regions. We demonstrate BrainSec is robust and has reliable performance by testing it on over 180 WSIs that incorporate numerous unique cases as well as distinct neuroanatomic brain regions. We also apply gradient-weighted class activation mapping (Grad-CAM) to interpret the segmentation masks and provide relevant explanations and insights. In addition, we have integrated BrainSec with an existing Amyloid-β pathology classification model into a unified framework (without incurring significant computation complexity) to identify pathologies, visualize their distributions, and quantify each type of pathologies in segmented GM/WM regions, respectively.

Patch-Based Cervical Cancer Segmentation using Distance from Boundary of Tissue.

Pathological diagnosis is used for examining cancer in detail, and its automation is in demand. To automatically segment each cancer area, a patch-based approach is usually used since a Whole Slide Image (WSI) is huge. However, this approach loses the global information needed to distinguish between classes. In this paper, we utilized the Distance from the Boundary of tissue (DfB), which is global information that can be extracted from the original image. We experimentally applied our method to the three-class classification of cervical cancer, and found that it improved the total performance compared with the conventional method.

A World-First Surgical Instrument for Minimally Invasive Robotically-Enabled Transplantation of Heart Patches for Myocardial Regeneration: A Brief Research Report.

Background: Patch-based approaches to regenerating damaged myocardium include epicardial surgical transplantation of heart patches. By the time this therapy is ready for widespread clinical use, it may be important that patches can be delivered via minimally invasive and robotic surgical approaches. This brief research report describes a world-first minimally invasive patch transplantation surgical device design enabled for human operation, master-slave, and fully automated robotic control.Method: Over a 12-month period (2019–20) in our multidisciplinary team we designed a surgical instrument to transplant heart patches to the epicardial surface. The device was designed for use via uni-portal or multi-portal Video-Assisted Thorascopic Surgery (VATS). For preliminary feasibility and sizing, we used a 3D printer to produce parts of a flexible resin model from a computer-aided design (CAD) software platform in preparation for more robust high-resolution metal manufacturing.Results: The instrument was designed as a sheath containing foldable arms, <2 cm in diameter when infolded to fit minimally invasive thoracic ports. The total length was 35 cm. When the arms were projected from the sheath, three moveable mechanical arms at the distal end were designed to hold a patch. Features included: a rotational head allowing for the arms to be angled in real time, a surface with micro-attachment points for patches and a releasing mechanism to release the patch.Conclusion: This brief research report represents a first step on a potential pathway towards minimally invasive robotic epicardial patch transplantation. For full feasibility testing, future proof-of-concept studies, and efficacy trials will be needed.

Nuclear atypia grading in breast cancer histopathological images based on CNN feature extraction and LSTM classification

Early diagnosis of breast cancer, the most common disease among women around the world, increases the chance of treatment and is highly important. Nuclear atypia grading in histopathological images plays an important role in the final diagnosis and grading of breast cancer. Grading images by pathologists is a time consuming and subjective task. Therefore, the existence of a computer-aided system for nuclear atypia grading is very useful and necessary. In this study, two automatic systems for grading nuclear atypia in breast cancer histopathological images based on deep learning methods are proposed. A patch-based approach is introduced due to the large size of the histopathological images and restriction of the training data. In the proposed system I, the most important patches in the image are detected first and then a three-hidden-layer convolutional neural network (CNN) is designed and trained for feature extraction and to classify the patches individually. The proposed system II is based on a combination of the CNN for feature extraction and a two-layer Long short-term memory (LSTM) network for classification. The LSTM network is utilised to consider all patches of an image simultaneously for image grading. The simulation results show the efficiency of the proposed systems for automatic nuclear atypia grading and outperform the current related studies in the literature.

Multi-label segmentation and detection of COVID-19 abnormalities from chest radiographs using deep learning

Due to COVID-19, demand for Chest Radiographs (CXRs) have increased exponentially. Therefore, we present a novel fully automatic modified Attention U-Net (CXAU-Net) multi-class segmentation deep model that can detect common findings of COVID-19 in CXR images. The architectural design of this model includes three novelties: first, an Attention U-net model with channel and spatial attention blocks is designed that precisely localize multiple pathologies; second, dilated convolution applied improves the sensitivity of the model to foreground pixels with additional receptive fields valuation, and third a newly proposed hybrid loss function combines both area and size information for optimizing model. The proposed model achieves average accuracy, DSC, and Jaccard index scores of 0.951, 0.993, 0.984, and 0.921, 0.985, 0.973 for image-based and patch-based approaches respectively for multi-class segmentation on Chest X-ray 14 dataset. Also, average DSC and Jaccard index scores of 0.998, 0.989 are achieved for binary-class segmentation on the Japanese Society of Radiological Technology (JSRT) CXR dataset. These results illustrate that the proposed model outperformed the state-of-the-art segmentation methods.

Intraoperative assessment of canine soft tissue sarcoma by deep learning enhanced optical coherence tomography.

Soft tissue sarcoma (STS) is a locally aggressive and infiltrative tumour in dogs. Surgical resection is the treatment of choice for local tumour control. Currently, post-operative pathology is performed for surgical margin assessment. Spectral-domain optical coherence tomography (OCT) has recently been evaluated for its value for surgical margin assessment in some tumour types in dogs. The purpose of this study was to develop an automatic diagnosis system that can assist clinicians in real-time for OCT image interpretation of tissues at surgical margins. We utilized a ResNet-50 network to classify healthy and cancerous tissues. A patch-based approach was adopted to achieve accurate classification with limited training data (80 cancer images, 80 normal images) and the validation set (20 cancer images, 20 normal images). The proposed method achieved an average accuracy of 97.1% with an excellent sensitivity of 94.3% on the validation set; the quadratic weighted κ was 0.94 for the STS diagnosis. In an independent test data set of 20 OCT images (10 cancer images, 10 normal images), the proposed method correctly differentiated all the STS images. Furthermore, we proposed a diagnostic curve, which could be evaluated in real-time to assist clinicians in detecting the specific location of a lesion. In short, the proposed method is accurate, operates in real-time and is non-invasive, which could be helpful for future surgical guidance.

Removing multi-frame Gaussian noise by combining patch-based filters with optical flow

Patch-based approaches such as 3D block matching (BM3D) and non-local Bayes (NLB) are widely accepted filters for removing Gaussian noise from single-frame images. In this work, we propose three extensions for these filters when there exist multiple frames of the same scene. The first of them employs reference patches on every frame instead of a commonly used single reference frame method, thus utilizing the complete available information. The remaining two techniques use a separable spatio-temporal filter to reduce interactions between dissimilar regions, hence mitigating artifacts. In order to deal with non-registered datasets we combine all our extensions with robust optical flow computation. Two of our proposed multi-frame filters outperform existing extensions on most occasions by a significant margin while also being competitive with a state-of-the-art neural network-based technique. Moreover, one of these two strategies is the fastest among all due to its separable design.