Computer-aided-diagnostic (CAD) techniques are of great help in facilitating the diagnosis of breast ultrasound (BUS) images. Conventional CAD approaches segment the region of interest (ROI), extract image features, and finally train a classifier for outputting the final diagnostic decision. Although some reported CAD systems have achieved high accuracy, their performance heavily depended on the source of training data. Meanwhile, the classification procedures formed by low-level features are hardly explicable. A novel CAD approach for BUS data with human-in-the-loop is therefore proposed in this paper. Differing from conventional CAD systems, user participation is involved to manually give the scores for selected Breast Imaging Reporting and Data System (BI-RADS) features. Three experienced radiologists obtained 1062 breast ultrasound data samples with BI-RADS feature scores, which formed a quantified score dataset for mining diagnostic patterns based on a constant column biclustering method. The discovered diagnostic patterns (i.e., the biclusters) are input into a fuzzy reasoning system to infer the final diagnostic decision (i.e., benign or malignant). In addition, a particle swarm optimization (PSO) algorithm is employed to optimize the parameters of the membership function and judgment threshold for the fuzzy reasoning. The proposed system skips the image processing by utilizing the artificial scoring datasets, and the BUS images from different sources can be classified. Meanwhile, more understandable and acceptable results can be given by the fuzzy reasoning. In the experiment, 10-fold cross-validation was used to evaluate system performance. The average accuracy reached 96.81%, specificity 95.45%, sensitivity 97.72%, positive predictive value (PPV) 97.03%, and negative predictive value (NPV) 96.57%, outperforming previously reported methods.