Cirrhosis is a liver disease resulting from abnormal continuation of fibrosis, and ultrasound imaging is widely used for cirrhosis diagnosis because of its non‐invasiveness. However, due to unclear appearances of cirrhosis on ultrasound images, diagnoses are difficult and individual results possibly differ depending on the physician's experience. Recently, computer‐aided diagnostic systems using image processing and machine learning have been developed to help physicians detect cirrhosis as a ‘Second opinion’. Some related studies have focused on a scenario where physicians set ROIs (Region of Interests) manually because selecting reliable ROIs for training a classifier and classification of patients is indispensable. But, the accuracy of such systems depends inherently on the quality of ROIs, and thus the workloads of physicians increase. In this paper, we propose a reliability evaluation method (REM) for each ROI based on its posterior probability and relationship to peripheral ROIs. The assumption of our proposal is that reliable regions of cirrhosis and normal can be observed in certain regions predominantly. We evaluated the effectiveness of the REM and its optimization for practical use. Experimental results showed that our proposed method curated reliable ROIs and improved classification performance in terms of AUC (Area Under the Curve). © 2024 Institute of Electrical Engineers of Japan and Wiley Periodicals LLC.