Histopathology currently acts as a gold standard for human prostate cancer (PCa) diagnosis. However, the subjective nature of histopathology leads to inevitable discordance among pathologists. Specifically, the inter-observer discordance could be up to 40% for the differentiation between Gleason score 6 (low-grade) and 7 (high-grade) of PCa. According to clinical guidelines, patients with high-grade PCa need to be actively treated, while patients with low-grade PCa could undergo active surveillance due to its slowly growing feature. Therefore, differentiating high-grade and low-grade PCa is an urgent clinical need. By integrating stimulated Raman scattering microscopy and confocal Raman spectroscopy, our previous study found the aberrant cholesteryl ester (CE) accumulation in human PCa tissues. However, no significant difference in CE accumulation between the low-grade and high-grade PCa was found, primarily because the previous study only analyzed the composition of manually selected lipid droplets (LDs) without quantitative analysis of the whole field of view. Here, we employed hyperspectral stimulated Raman scattering (HSRS) microscopy to test the hypothesis of CE as a marker for differentiation of low-grade and high-grade human PCa. First, lipid, lipofuscin, and protein were quantitatively mapped in human prostate tissues based on HSRS imaging of C–H vibrational region and multivariate curve resolution analysis. Then, within the lipid channel, CE percentage and unsaturation level of LDs were quantitatively mapped according to the height ratio between Raman bands at 2870 and 2850 cm−1, and between Raman bands at 3006 and 2850 cm−1, respectively. In total of 6 normal prostate tissues, 9 low-grade and 9 high-grade PCa tissues from patients, we found lipofuscin accumulation in all the normal prostate but none in PCa. While all the high-grade PCa tissues had LD accumulation, only 3 low-grade PCa tissues had LD accumulation. Notably, among all the PCa tissues with LD accumulation, no significant difference in LD amount was found between low-grade and high-grade PCa. Fortunately, both CE percentage and unsaturation level of the LDs were significantly different between low-grade and high-grade PCa. Furthermore, it was shown that CE percentage could differentiate low-grade and high-grade PCa with high sensitivity and specificity. Taken together, our study may provide a new opportunity towards more accurate PCa diagnosis.