Colorectal cancer diagnosis currently relies on histological detection of endoluminal neoplasia in biopsy specimens. However, clinical visual endoscopy provides no quantitative subsurface cancer information. In this ex vivo study of nine fresh human colon specimens, we report the first use of quantified subsurface scattering coefficient maps acquired by swept-source optical coherence tomography to reveal subsurface abnormities. We generate subsurface scattering coefficient maps with a novel wavelet-based-curve-fitting method that provides significantly improved accuracy. The angular spectra of scattering coefficient maps of normal tissues exhibit a spatial feature distinct from those of abnormal tissues. An angular spectrum index to quantify the differences between the normal and abnormal tissues is derived, and its strength in revealing subsurface cancer in ex vivo samples is statistically analyzed. The study demonstrates that the angular spectrum of the scattering coefficient map can effectively reveal subsurface colorectal cancer and potentially provide a fast and more accurate diagnosis.