Reading mammograms is a difficult task and for this reason any development that may improve the performance in breast cancer screening is of great importance. We proposed optimized computer aided diagnosis (CAD) system, equipped with reliability estimate module, for mass detection on digitized mammograms. Proposed CAD system consists of four major steps: preprocessing, segmentation, feature extraction and classification. We propose a simple regression function as a threshold function for extraction of potential masses. By running optimization procedure we estimate parameters of the preprocessing and segmentation steps thus ensuring maximum mass detection sensitivity. In addition to the classification, where we tested seven different classifiers, the CAD system is equipped with reliability estimate module. By performing segmentation 91.3% of masses were correctly segmented with 4.14 false positives per image (FPpi). This result is improved in the classification phase where, among the seven tested classifiers, multilayer perceptron neural network achieved the best result including 77.4% sensitivity and 0.49 FPpi. By using the proposed regression function and parameter optimization we were able to improve segmentation results comparing to the literature. In addition, we showed that CAD system has high potential for being equipped with reliability estimate module.