Current breast tumor margin detection methods are destructive, time-consuming, and result in significant reoperative rates. Dual-modality photoacoustic tomography (PAT) and ultrasound has the potential to enhance breast margin characterization by providing clinically relevant compositional information with high sensitivity and tissue penetration. However, quantitative methods that rigorously compare volumetric PAT and ultrasound images with gold-standard histology are lacking, thus limiting clinical validation and translation. Here, we present a quantitative multimodality workflow that uses inverted Selective Plane Illumination Microscopy (iSPIM) to facilitate image co-registration between volumetric PAT-ultrasound datasets with histology in human invasive ductal carcinoma breast tissue samples. Our ultrasound-PAT system consisted of a tunable Nd:YAG laser coupled with a 40 MHz central frequency ultrasound transducer. A linear stepper motor was used to acquire volumetric PAT and ultrasound breast biopsy datasets using 1100 nm light to identify hemoglobin-rich regions and 1210 nm light to identify lipid-rich regions. Our iSPIM system used 488 nm and 647 nm laser excitation combined with Eosin and DRAQ5, a cell-permeant nucleic acid binding dye, to produce high-resolution volumetric datasets comparable to histology. Image thresholding was applied to PAT and iSPIM images to extract, quantify, and topologically visualize breast biopsy lipid, stroma, hemoglobin, and nuclei distribution. Our lipid-weighted PAT and iSPIM images suggest that low lipid regions strongly correlate with malignant breast tissue. Hemoglobin-weighted PAT images, however, correlated poorly with cancerous regions determined by histology and interpreted by a board-certified pathologist. Nuclei-weighted iSPIM images revealed similar cellular content in cancerous and non-cancerous tissues, suggesting malignant cell migration from the breast ducts to the surrounding tissues. We demonstrate the utility of our nondestructive, volumetric, region-based quantitative method for comprehensive validation of 3D tomographic imaging methods suitable for bedside tumor margin detection.