Breast cancer is one of the serious health issues which are quite frequent among women and the mortality rate due to it can be reduced if detected and diagnosed at the early stage. In this paper, a new generalized deep learning architecture, deep dilated fully convolutional neural network (DDFCNN), is designed to detect multiple abnormalities like mass, microcalcification, and architectural distortion in mammograms. The proposed DDFCNN architecture consists of a feature and a dilation module. In feature module, convolutional kernels of various sizes are used to capture information at different scales. Since there is no loss of resolution in the output of the dilation module, it renders more details which helps in localizing the anomaly properly. Several experiments are conducted on a widely used databases — DDSM and mini-MIAS where an accuracy of 95.33%, 91.42%, and 92.67% is measured for detection of mass, microcalcification, and architectural distortion, respectively with a FPs/I of 0.43, 0.50, and 0.46 for DDSM and similar assessment for mini-MIAS database are 96.56%, 89.63%, and 95.07% with a FPs/I of 0.29, 0.51, and 0.31. Moreover, the overall performance in detection of all the abnormalities in combination is 93.14% with 0.46 as FPs/I for DDSM whereas for mini-MIAS is 93.75% with 0.37 as FPs/I. The performance is further compared with the state-of-the-other methods where the proposed approach takes an edge.