Hard Pixels Research Articles

Hardness-aware loss for object segmentation

In object segmentation, the existence of hard-classified-pixels limits the segmentation performance. Focusing on these hard pixels through assigning different weights to pixel loss can guide the learning of segmentation model effectively. Existing loss weight assignment methods perceive pixels hardness by current predicted information, pay less attention to past predicted information. While current studies show that samples with less improvement in predicted probability compared to the past are difficult to learn. To define hard pixels more accurately, a hardness-aware loss for object segmentation is proposed. Firstly, the metric of pixel hardness degree is defined, and a mapping function is proposed to quantitatively evaluate the hardness degree which is defined on the difference between current and past predicted probabilities. Then a new compound metric, hardness value, is defined based on hardness degree and the uncertainty. Based on the compound metric, a new loss function is proposed. Experiment results on four datasets using convolutional neural network and Transformer as the backbone models demonstrate that the proposed method effectively improves the accuracy of object segmentation. Especially, in the segmentation model based on ResNet-50, the proposed method improves mean Intersection over Union (mIoU) by almost 4.3 % compared to cross entropy on DUT-O dataset.

Separate first, then segment: An integrity segmentation network for salient object detection

Current methods aggregate multi-level features or introduce auxiliary information to get more refined saliency maps. However, little attention is paid to how to obtain complete salient objects in cluttered background. To address this problem, we propose an integrity segmentation network (ISNet) with a novel detection paradigm that first separates the targets completely and then segment them finely. Specifically, the ISNet architecture consists of a target separation (TS) branch and an object segmentation (OS) branch, trained using a hierarchical difference-aware (HDA) loss. The TS branch equipped with a fractal structure is utilized to produce saliency features with expanded boundary (SF w/ EB), which can enlarge the difference of border details to separate the target from background completely. Compared with the edge and skeleton information, the SF w/ EB contains a more complete structure, which can supplement the defect of salient objects. The OS branch is leveraged to generate complementary features, which gradually integrates the SF w/ EB and aggregated features to segment complete saliency maps. Moreover, we propose the HDA loss to further improve the structural integrity of prediction, which hierarchically assigns weight to pixels according to their differences. Hard pixels will be given more attention to discriminate the similar parts between foreground and background. Comprehensive experimental results on five datasets show that the proposed ISNet outperforms the state-of-the-art methods both quantitatively and qualitatively. Concretely, compared with three typical models, the average gain percentage reaches 2.6% in terms of Fβ, Sm and MAE on two large complex datasets. The improvements demonstrate that the proposed ISNet are beneficial for improving the integrity of prediction. Besides, the ISNet is efficient and runs at a real-time speed of 39.5 FPS when processing an image with size of 320 × 320. Furthermore, the proposed model has better generalization, which can also be applied to other vision tasks to handle complex scenes. Codes are available at https://github.com/lesonly/ISNet.

CMGNet: Context-aware middle-layer guidance network for salient object detection

Salient object detection (SOD) is a critical task in computer vision that involves accurately identifying and segmenting visually significant objects in an image. To address the challenges of gridding issues and feature dilution effects commonly encountered in SOD, we propose a sophisticated context-aware middle-layer guidance network (CMGNet). CMGNet incorporates the context-aware central-layer guidance module (CCGM), which utilizes cost-effective large kernels of depth-wise convolutions with embedded parallel channel attentions and squeeze-and-excitation (SeE) attentions mechanisms. It enables the model to effectively perceive objects of varying scales in complex scenarios. Additionally, the incorporation of the adjacent-to-central-layers paradigm enriches the model’s ability to capture more structural and contextual information. To further enhance performance, we introduce the dual-phase central-layer refinement module (DCRM), which effectively removes the minute blurry residuals in complex scenarios and enhances object segmentation. Moreover, we propose a novel hybrid loss function that handles hard pixels at or near boundaries by incorporating a weighting formula. This hybrid loss function combines binary cross-entropy (BCE), intersection over union (IoU), and consistency-enhanced loss (CEL), resulting in smoother and more precise saliency maps. Extensive evaluations on challenging datasets demonstrate the superiority of our approach over 15 state-of-the-art methods in salient object detection.

A multiscale bidirectional fuzzy-driven learning network for remote sensing image segmentation

ABSTRACT Semantic segmentation is a fundamental but meaningful task in the remote sensing image understanding community. Great progress has been made in optical sensor photography technology, which poses an opportunity and a challenge for remote sensing image segmentation task. But, in fact, a longstanding and intractable problem is that many hard pixels in special position, i.e. the prevalent intra-class noise and a poor boundary delineation, is difficult to classify due to their inherent uncertainty. In this paper, we comprehensively consider the characteristics of deep learning and introduce traditional pattern recognition methods to drive structure learning, which can leverage the corresponding fuzzy logic model to alleviate the aforementioned problem in remote sensing images. Specifically, this paper designs a multiscale bidirectional fuzzy-driven learning network (MBFNet), which takes advantage of both deep learning and fuzzy logic to effectively alleviate the inherent uncertainty of these hard pixels. The structure of convolutional neural networks driven by fuzzy systems also provides a new modelling paradigm for solving the uncertain problem in remote sensing images. Meanwhile, multiscale techniques and bidirectional fusion are introduced to enhance feature aggregation and avoid the potential adverse effects of fuzzy systems, respectively. Experimental results on two datasets demonstrate qualitatively and quantitatively that the proposed MBFNet is competitive.

Key information-guided networks for medical image segmentation in medical systems

Motivated by the goals of overcoming inherent engineering difficulties on segmentation accuracy arising from low contrast and complex tissue relationships in medical images, this paper investigates how key information can be extracted and analyzed in a targeted manner to discover relationships between tissues and enhance feature differentiation. It proposes the Key Information-guided segmentation algorithm, which consists of two key components: the Key Region-guided Sparse Self-Attention (KRSSA) module and the Hard Pixels Multisubclass Contrast learning (HPMCL) module. The former is applied in the encoder by activating the importance of global features and adaptively selecting attentional regions. Experimental results show that propagating large amounts of complex and redundant information suppresses learning. The latter is applied in the final decision layer, where the feature space of each class is partitioned into several subclasses, and hard pixels are continuously pulled towards the closest subclass of their respective class, away from all subclasses of other classes by contrastive learning. Experimental results show that moving pixels towards subclasses rather than a single class center leads to more effective segmentation results. In summary, we propose a novel visual model and extensively validate it on the Brain Tumor Segmentation 2019 Challenge (BraTS2019) dataset, BraTS2020 dataset, and the Medical Segmentation Decathlon (MSD) dataset, namely in brain tumors and the hippocampus. The Dice similarity coefficients of 0.806, 0.904, and 0.853 for the enhanced tumor, whole tumor, and tumor core, respectively, in the Brats2020 validation set ranked fourth out of 345 participating methods. This method has significant implications for addressing challenges in medical image segmentation, including low attention to key regions, class imbalance, and background interference.

Supplement and Suppression: Both Boundary and Nonboundary are Helpful for Salient Object Detection.

Current methods aggregate multilevel features from the backbone and introduce edge information to get more refined saliency maps. However, little attention is paid to how to suppress the regions with similar saliency appearances in the background. These regions usually exist in the vicinity of salient objects and have high contrast with the background, which is easy to be misclassified as foreground. To solve this problem, we propose a gated feature interaction network (GFINet) to integrate multiple saliency features, which can utilize nonboundary features with background information to suppress pseudosalient objects and simultaneously apply boundary features to supplement edge details. Different from previous methods that only consider the complementarity between saliency and boundary, the proposed network introduces nonboundary features into the decoder to filter the pseudosalient objects. Specifically, GFINet consists of global features aggregation branch (GFAB), boundary and nonboundary features' perception branch (B&NFPB), and gated feature interaction module (GFIM). According to the global features generated by GFAB, boundary and nonboundary features produced by B&NFPB and GFIM employ a gate structure to adaptively optimize the saliency information interchange between abovementioned features and, thus, predict the final saliency maps. Besides, due to the imbalanced distribution between the boundary pixels and nonboundary ones, the binary cross-entropy (BCE) loss is difficult to predict the pixels near the boundary. Therefore, we design a border region aware (BRA) loss to further boost the quality of boundary and nonboundary, which can guide the network to focus more on the hard pixels near the boundary by assigning different weights to different positions. Compared with 12 counterparts, experimental results on five benchmark datasets show that our method has better generalization and improves the state-of-the-art approach by 4.85% averagely in terms of the regional and boundary evaluation measures. In addition, our model is more efficient with an inference speed of 50.3 FPS when processing a 320 x 320 image. Code has been made available at https://github.com/lesonly/GFINet.

Dynamic contrastive learning guided by class confidence and confusion degree for medical image segmentation

This work proposes an intra-Class-confidence and inter-Class-confusion guided Dynamic Contrastive (CCDC) learning framework for medical image segmentation. A core contribution is to dynamically select the most expressive pixels to build positive and negative pairs for contrastive learning at different training phases. For the positive pairs, dynamically adaptive sampling strategies are introduced for sampling different sets of pixels based on their hardness (namely the easiest, easy, and hard pixels). For the negative pairs, to efficiently learn from the classes with high confusion degree w.r.t a query class (i.e., a class containing the query pixels), a new hard class mining strategy is presented. Furthermore, pixel-level representations are extended to the neighbourhood region to leverage the spatial consistency of adjacent pixels. Extensive experiments on the three public datasets demonstrate that the proposed method significantly surpasses the state-of-the-art.

Positive–negative equal contrastive loss for semantic segmentation

The contextual information is critical for various computer vision tasks, previous works commonly design plug-and-play modules and structural losses to effectively extract and aggregate the global context. These methods utilize fine-label to optimize the model but ignore that fine-trained features are also precious training resources, which can introduce preferable distribution to hard pixels (i.e., misclassified pixels). Inspired by contrastive learning in unsupervised paradigm, we apply the contrastive loss in a supervised manner and re-design the loss function to cast off the stereotype of unsupervised learning (e.g., imbalance of positives and negatives, confusion of anchors computing). To this end, we propose Positive-Negative Equal contrastive loss (PNE loss), which increases the latent impact of positive embedding on the anchor and treats the positive as well as negative sample pairs equally. The PNE loss can be directly plugged right into existing semantic segmentation frameworks and leads to excellent performance with neglectable extra computational costs. We utilize a number of classic segmentation methods (e.g., DeepLabV3, HRNetV2, OCRNet, UperNet) and backbone (e.g., ResNet, HRNet, Swin Transformer) to conduct comprehensive experiments and achieve state-of-the-art performance on three benchmark datasets (e.g., Cityscapes, COCO-Stuff and ADE20K). Our code will be publicly available at https://github.com/jingw193/PNE_Loss.

SDDet: An Enhanced Encoder–Decoder Network With Hierarchical Supervision for Surface Defect Detection

Surface defect detection (SDDet) is extremely critical for quality control and routine maintenance, and computer vision-based methods have delivered promising performance in various industrial fields. However, there remains big challenges in region consistency and boundary localization, due to complicated appearance in defects, interference of artifacts, and low contrast. To overcome these issues, in this article, we propose an enhanced encoder–decoder network with hierarchical supervision for SDDet. Specifically, deformable convolution is adopted to enable feature learning in a free-form way, thus making the network more robust to various defect structures. Then, a bridge module is applied to capture richer multilevel contextual information, which is conducive to distinguishing defects from the background artifacts. Next, the learned features from adjacent layers are adaptively combined by selective feature aggregation (SFA) module to enhance discriminative features and suppress valueless features. Meanwhile, the bottleneck refinement (BR) module is added to further refine the boundary of defect region, resulting in a satisfying boundary details. Finally, a hybrid three-level loss is developed to guide the network to focus more on error-prone hard pixels. Additionally, a hardware system embedded with our SDDet is built to achieve automatic and accurate defect inspection. Experimental results over four datasets demonstrate the effectiveness and generalization of our method compared with state-of-the-art methods.

Segmentation of ten fetal heart components with coarse‐to‐fine cascading and dynamic feature powering

Segmenting heart components in the apical four-chamber view of fetal echocardiography is of critical significance in clinical practice. However, it is difficult to recognize these components due to small-scale components and the imbalanced ventricular apex orientation. In this study, a novel segmentation framework is proposed to segment ten general fetal heart components for the first time. This framework consists of a multi-directional fine-density (MDFD) data augmentation method and a coarse-to-fine cascade network (CFCN). MDFD enhances the apex orientation diversity and balances the orientation distribution. CFCN has two stages including a coarse network and a fine network. These two stages have similar structures that consist of a feature extractor and a feature refined layer named as Element-Wise Power with Dynamic Exponent layer (EWPDE). EWPDE which is a plug-and-play module for segmentation refines the features from the feature extractor to position small components accurately. By adopting EWPDE, the influence of each pixel is adjusted and hard pixels of small components are segmented precisely. Based on the dataset, the method is proved to be effective with the high mean intersection over union (mIoU) value and low missing ratio (MR). With MDFD and EWPDE, CFCN that adopts DeepLabV3+ as the feature extractor outperforms the best segmentation results (mIoU:0.480, MR:0.035). Compared to the original performance (mIoU:0.407, MR:0.085) of DeepLabV3+, the method improves the results significantly.

Self-Paced Knowledge Distillation for Real-Time Image Guided Depth Completion

Image guided depth completion aims to generate a dense depth map from a sparse one with the guidance of a color image. Previous high-accuracy methods often rely on complex networks that are large in size and expensive in computational cost, making them inapplicable to real-time platforms. In this letter, we propose a self-paced knowledge distillation method, which obtains a lightweight but accurate depth completion model via distilling knowledge from a complex teacher network. Specifically, by taking advantage of the easy-to-hard learning curriculum in deep networks, we first design a groundtruth-free hard-pixel mining module to tell hard and noisy pixels in the teacher’s output. Then, we design two self-paced distillation losses, which gradually introduce hard pixels to distill the depth and structure knowledge from the teacher to the compact student network. Experiments on the KITTI benchmark show that the proposed method can improve the original student model by a considerable margin. The distilled compact and real-time student model outperforms all previous lightweight networks, mitigating the performance gap with state-of-the-art high-accuracy but complex models.

Multi-granularity scale-aware networks for hard pixels segmentation of pulmonary nodules

Accurate automatic segmentation of pulmonary nodules can greatly assist in the early clinical diagnosis and analysis of lung cancer. However, it remains a challenging task due to (1) the variable scales, complex shapes, and textures heterogeneity of nodules and (2) the similar visual characteristics of nodules and the surrounding environment. These factors make it difficult for a model to capture the representative and distinguishing features of nodules. We define these pixels, which are difficult to segment accurately, as “hard pixels.” To comprehensively tackle these challenges, we focus on the complementarity between surrounding background information and salient nodules information. Accordingly, we propose a Multi-Granularity Scale-Aware Networks (MGSA-Net) for accurate pulmonary nodules segmentation with steps as follows. First, to effectively preserve both global contextual and local fine details information, we unify the representation of the feature about global and patch-level images in a single framework. Second, we introduce a deep scale-aware module (DSAM) in the global stream that could generate multi-scale feature maps with a uniform representational power, which can process more contextual information. Finally, we propose a multi-granularity feature map sharing learning to fuse feature maps from the dual branch at various scales. Benefiting from the rich background information and salient nodules information, the fused features can help simultaneously capture the similarity of nodules and the diversity of background to mine hard pixels. Especially, the two streams are jointly optimized, ensuring they are mutually reinforced and refining current pixels’ predictions by their similar structure boundaries. Extensive experiments on the LIDC-IDRI dataset have demonstrated that the proposed MGSA-Net could surpass most segmentation models and advance the state-of-the-art performance with the dice similarity coefficient (DSC) of 87.32% on the dataset. Code can be available at: https://github.com/ISSE-AILab/MGSA-Net.

To Choose or to Fuse? Scale Selection for Crowd Counting

In this paper, we address the large scale variation problem in crowd counting by taking full advantage of the multi-scale feature representations in a multi-level network. We implement such an idea by keeping the counting error of a patch as small as possible with a proper feature level selection strategy, since a specific feature level tends to perform better for a certain range of scales. However, without scale annotations, it is sub-optimal and error-prone to manually assign the predictions for heads of different scales to specific feature levels. Therefore, we propose a Scale-Adaptive Selection Network (SASNet), which automatically learns the internal correspondence between the scales and the feature levels. Instead of directly using the predictions from the most appropriate feature level as the final estimation, our SASNet also considers the predictions from other feature levels via weighted average, which helps to mitigate the gap between discrete feature levels and continuous scale variation. Since the heads in a local patch share roughly a same scale, we conduct the adaptive selection strategy in a patch-wise style. However, pixels within a patch contribute different counting errors due to the various difficulty degrees of learning. Thus, we further propose a Pyramid Region Awareness Loss (PRA Loss) to recursively select the most hard sub-regions within a patch until reaching the pixel level. With awareness of whether the parent patch is over-estimated or under-estimated, the fine-grained optimization with the PRA Loss for these region-aware hard pixels helps to alleviate the inconsistency problem between training target and evaluation metric. The state-of-the-art results on four datasets demonstrate the superiority of our approach. The code will be available at: https://github.com/TencentYoutuResearch/CrowdCounting-SASNet.

Hard Pixel Mining for Depth Privileged Semantic Segmentation

Semantic segmentation has achieved remarkable progress but remains challenging due to the complex scene, object occlusion, and so on. Some research works have attempted to use extra information such as a depth map to help RGB based semantic segmentation because the depth map could provide complementary geometric cues. However, due to the inaccessibility of depth sensors, depth information is usually unavailable for the test images. In this paper, we leverage only the depth of training images as the privileged information to mine the hard pixels in semantic segmentation, in which depth information is only available for training images but not available for test images. Specifically, we propose a novel Loss Weight Module, which outputs a loss weight map by employing two depth-related measurements of hard pixels: Depth Prediction Error and Depth-aware Segmentation Error. The loss weight map is then applied to segmentation loss, with the goal of learning a more robust model by paying more attention to the hard pixels. Besides, we also explore a curriculum learning strategy based on the loss weight map. Meanwhile, to fully mine the hard pixels on different scales, we apply our loss weight module to multi-scale side outputs. Our hard pixels mining method achieves the state-of-the-art results on three benchmark datasets, and even outperforms the methods which need depth input during testing.

NPALOSS: NEIGHBORING PIXEL AFFINITY LOSS FOR SEMANTIC SEGMENTATION IN HIGH-RESOLUTION AERIAL IMAGERY

Abstract. The performance of semantic segmentation in high-resolution aerial imagery has been improved rapidly through the introduction of deep fully convolutional neural network (FCN). However, due to the complexity of object shapes and sizes, the labeling accuracy of small-sized objects and object boundaries still need to be improved. In this paper, we propose a neighboring pixel affinity loss (NPALoss) to improve the segmentation performance of these hard pixels. Specifically, we address the issues of how to determine the classifying difficulty of one pixel and how to get the suitable weight margin between well-classified pixels and hard pixels. Firstly, we convert the first problem into a problem that the pixel categories in the neighborhood are the same or different. Based on this idea, we build a neighboring pixel affinity map by counting the pixel-pair relationships for each pixel in the search region. Secondly, we investigate different weight transformation strategies for the affinity map to explore the suitable weight margin and avoid gradient overflow. The logarithm compression strategy is better than the normalization strategy, especially the common logarithm. Finally, combining the affinity map and logarithm compression strategy, we build NPALoss to adaptively assign different weights for each pixel. Comparative experiments are conducted on the ISPRS Vaihingen dataset and several commonly-used state-of-the-art networks. We demonstrate that our proposed approach can achieve promising results.

Two-stream deep sparse network for accurate and efficient image restoration

Deep convolutional neural network (CNN) has achieved great success in image restoration. However, previous methods ignore the complementarity between low-level and high-level features, thereby leading to limited image reconstruction quality. In this paper, we propose a two-stream sparse network (TSSN) to explicitly learn shallow and deep features to enforce their respective contribution to image restoration. The shallow stream learns shallow features (e.g., texture edges), and the deep stream learns deep features (e.g., salient semantics). In each stream, sparse residual block (SRB) is proposed to efficiently aggregate hierarchical features by constructing sparse connections among layers in the local block. Spatial-wise and channel-wise attention are used to fuse the shallow and deep stream which recalibrates features by weight assignment in both spatial and channel dimensions. A novel loss function called Softmax-L1 loss is proposed to increase penalties of pixels that have large L1 loss (i.e., hard pixels). TSSN is evaluated with three representative IR applications, i.e., single image super-resolution, image denoising and JPEG deblocking. Extensive experiments demonstrate that TSSN outperforms most of state-of-the-art methods on benchmark datasets on both quantitative metric and visual quality.

F³Net: Fusion, Feedback and Focus for Salient Object Detection

Most of existing salient object detection models have achieved great progress by aggregating multi-level features extracted from convolutional neural networks. However, because of the different receptive fields of different convolutional layers, there exists big differences between features generated by these layers. Common feature fusion strategies (addition or concatenation) ignore these differences and may cause suboptimal solutions. In this paper, we propose the F3Net to solve above problem, which mainly consists of cross feature module (CFM) and cascaded feedback decoder (CFD) trained by minimizing a new pixel position aware loss (PPA). Specifically, CFM aims to selectively aggregate multi-level features. Different from addition and concatenation, CFM adaptively selects complementary components from input features before fusion, which can effectively avoid introducing too much redundant information that may destroy the original features. Besides, CFD adopts a multi-stage feedback mechanism, where features closed to supervision will be introduced to the output of previous layers to supplement them and eliminate the differences between features. These refined features will go through multiple similar iterations before generating the final saliency maps. Furthermore, different from binary cross entropy, the proposed PPA loss doesn't treat pixels equally, which can synthesize the local structure information of a pixel to guide the network to focus more on local details. Hard pixels from boundaries or error-prone parts will be given more attention to emphasize their importance. F3Net is able to segment salient object regions accurately and provide clear local details. Comprehensive experiments on five benchmark datasets demonstrate that F3Net outperforms state-of-the-art approaches on six evaluation metrics. Code will be released at https://github.com/weijun88/F3Net.

M-Net: A Novel U-Net With Multi-Stream Feature Fusion and Multi-Scale Dilated Convolutions for Bile Ducts and Hepatolith Segmentation

Automatically segmenting bile ducts and hepatolith in abdominal CT scans is helpful to assist hepatobiliary surgeons for minimally invasive surgery. High-deformation characteristics of bile ducts and small-size characteristics of hepatolith make this segmentation task challenging. To the best of our knowledge, we make the first attempt to simultaneously segment bile ducts and hepatolith in this paper. Inspired by U-Net, a novel two-dimensional end-to-end fully convolutional network named M-Net is designed to implement this segmentation task. The M-Net is composed of four streams involving two encoder-decoder processes. Multi-scale dilated convolutions are designed to extract abundant semantic features and multi-scale context information at different scales. To make full advantages of multi-scale feature maps, a multi-stream feature fusion strategy is proposed to transfer the most abundant semantic features produced in the first stream to the other streams. To further improve the segmentation performance, a novel loss function is defined to focus the M-Net on hard pixels (difficultly distinguished) in the edges of bile ducts and hepatolith, which is based on the online bootstrapped method and cross entropy. By discarding pixels (easy to distinguish) with higher probability of class, the decline of loss is focused on hard pixels so that the training become more efficient and directional. Experimental results indicate that our proposed M-Net is superior to the state-of-the-art deep-learning methods for simultaneously segmenting bile ducts and hepatolith in the abdominal CT scans. The M-Net can simultaneously segment bile ducts and hepatolith in abdominal CT scans at a high performance with 98.678% Recall, 84.427% Precision, 89.831% DICE and 90.998% F1-score for bile ducts, and 99.894% Recall, 55.132% Precision, 71.248% DICE and 71.051% F1-score for hepatolith.

An End-to-End Neural Network for Road Extraction From Remote Sensing Imagery by Multiple Feature Pyramid Network

Unlike single geospatial objects extraction from high-resolution remote sensing images, the task of road extraction faces more challenges, including its narrowness, sparsity, diversity, multiscale characteristics, and class imbalance. Focusing on these challenges, this paper proposes an end-to-end framework called the multiple feature pyramid network (MFPN). In MFPN, we design an effective feature pyramid and a tailored pyramid pooling module, taking advantage of multilevel semantic features of high-resolution remote sensing images. In the optimization stage, a weighted balance loss function is presented to solve the class imbalance problem caused by the sparseness of roads. The proposed novel loss function is more sensitive to the misclassified and the sparse real labeled pixels and helps to focus on the spare set of hard pixels in the training stage. Compared with the cross-entropy loss function, the weighted balance loss can reduce training time dramatically for the same precision. Experiments on two challenging datasets of high-resolution remote sensing images which illustrate the performance of the proposed algorithm have achieved significant improvements, especially for narrow rural roads.

Automatic Selection of Representative Slice From Cine-Loops of Real-Time Sonoelastography for Classifying Solid Breast Masses

This study aimed to evaluate the performance of automatic selection of representative slice from cine-loops of real-time sonoelastography for classifying benign and malignant breast masses. This retrospective study included 141 ultrasound elastographic studies (93 benign and 48 malignant masses). A novel computer-assisted system was developed for the automatic segmentation of the targeted lesion from cine-loops of real-time sonoelastography. Its hard ratio, defined as the ratio of the number of hard pixels within the tumor divided by the total number of pixels of the whole tumor, was also calculated. The targeted mass was segmented by edge-detection and region growing methods, with combined motion registration after manually defining the original seed. Signal-to-noise ratio ( SNR e ) and contrast-to-noise ratio ( CNR e ) of ultrasound elastogram were computed to obtain an optimum slice for differentiating benign and malignant lesions. The diagnostic results of automatic slice selection using maximum strain, maximum SNR e , maximum CNR e , maximum compression and the slices selected by radiologists were compared. Mann–Whitney U test, performance indexes and receiver operating characteristic (ROC) curves were used for statistical analysis. Performance using the maximum SNR e (accuracy 84.4%, sensitivity 83.3%, specificity 85.0% and A z value 0.90) was the best as compared with those of maximum CNR e (82.3%, 79.2%, 83.9% and 0.88, respectively), maximum compression (78.0%, 79.2%, 77.4% and 0.85, respectively), maximum strain (79.4%, 79.2%, 79.6% and 0.87, respectively) and radiologists’ selection (77.3%, 77.1%, 77.4% and 0.80, respectively). Automatic selection of representative slice from the cine-loops of real-time sonoelastography is a practical, objective and accurate approach for classifying solid breast masses.(E-mail: moonwk@radcom.snu.ac.kr and rfchang@csie.ntu.edu.tw)