The College of American Pathologists (CAP, Northfield, Illinois) monitors performance in cytologic analysis to evaluate the standard of practice and consider strategies for method improvement. 5700 responses to 97 pancreatobiliary tract brushing slide challenges were collected by the CAP Non-Gynecologic Cytopathology (NGC) Program, between 2000 and 2011. Analysis examined participant agreement with the general diagnostic categories of benign or malignant. Suspicious responses were classified as concordant with slides having a positive general diagnosis. Conventional smears with Pap stain and Romanowsky stain were evaluated in addition to CytoSpin, ThinPrep, and SurePath preparations. A nonlinear mixed model was fit with 3 factors-general diagnosis, participant type, and preparation type. Overall concordance rate was 91.7%. Preparation type and general diagnosis were significantly associated with the concordance rate. The interaction term between these two factors was also statistically significant, with ThinPrep performing marginally better for positive cases and CytoSpin performing better for negative cases. Conventional smears did not perform as well as CytoSpin, ThinPrep, or SurePath. Participants performed well with greater than 90% agreement with the target diagnostic category. There was no significant difference between cytotechnologists and pathologists. Small significant differences were found between preparations types. The statistical differences between concentration techniques may be due to dissimilarities in the quantity of cells and quality of cytomorphology, thus affecting the interpretations by participating laboratories.