Early detection leading to timely treatment in the initial stages of cancer may decrease the breast cancer death rate. We propose deep learning techniques along with image processing for the detection of tumors. The availability of online datasets and advances in graphical processing units (GPU) have promoted the application of deep learning models for the detection of breast cancer. In this paper, deep learning models using convolutional neural network (CNN) have been built to automatically classify mammograms into benign and malignant. Issues like overfitting and dataset imbalance are overcome. Experimentation has been done on two publicly available datasets, namely mammographic image analysis society (MIAS) database and digital database for screening mammography (DDSM). Robustness of the models is accomplished by merging the datasets. In our experimentation, MatConvNet has achieved an accuracy of 94.2% on the merged dataset, performing the best amongst all the CNN models used individually. Hungarian optimization algorithm is employed for selection of individual CNN models to form an ensemble. Ensemble of CNN models led to an improved performance, resulting in an accuracy of 95.7%.