Beam-quality optimization in digital mammography traditionally considers detection of a target obscured by quantum noise in a homogeneous background. This does not correspond well to the clinical imaging task because real mammographic images contain a complex superposition of anatomical structures, resulting in anatomical noise that may dominate over quantum noise. The purpose of this paper is to assess the influence on optimal beam quality in mammography when anatomical noise is taken into account. The detectability of microcalcifications and masses was quantified using a theoretical ideal-observer model that included quantum noise as well as anatomical noise and a simplified model of a photon-counting mammography system. The outcome was experimentally verified using two types of simulated tissue phantoms. The theoretical model showed that the detectability of tumors and microcalcifications behaves differently with respect to beam quality and dose. The results for small microcalcifications were similar to what traditional optimization methods yield, which is to be expected because quantum noise dominates over anatomical noise at high spatial frequencies. For larger tumors, however, low-frequency anatomical noise was the limiting factor. Because anatomical structure noise has similar energy dependence as tumor contrast, the optimal x-ray energy was found to be higher and the useful energy region was wider than traditional methods suggest. A simplified scalar model was able to capture this behavior using a fitted noise mixing parameter. The phantom measurements confirmed these theoretical results. It was shown that since quantum noise constitutes only a small fraction of the noise, the dose could be reduced substantially without sacrificing tumor detectability. Furthermore, when anatomical noise is included, the tube voltage can be increased well beyond what is conventionally considered optimal and used clinically, without loss of image quality. However, no such conclusions can be drawn for the more complex mammographic imaging task as a whole.