Collagen and lipid are important components of tumor microenvironments (TME) and participates in tumor development and invasion. It has been reported that collagen and lipid can be used as a hallmark to diagnosis and differentiate tumors. We aim to introduce photoacoustic spectral analysis (PASA) method that can provide both the content and structure distribution of endogenous chromophores in biological tissues to characterize the tumor-related features for identifying different types of tumors. Ex vivo human tissues with suspected squamous cell carcinoma (SCC), suspected basal cell carcinoma (BCC), and normal tissue were used in this study. The relative lipid and collagen contents in the TME were assessed based on the PASA parameters and compared with histology. Support vector machine (SVM), one of the simplest machine learning tools, was applied for automatic skin cancer type detection. The PASA results showed that the lipid and collagen levels of the tumors were significantly lower than those of the normal tissue, and there was a statistical difference between SCC and BCC (), consistent with the histopathological results. The SVM-based categorization achieved diagnostic accuracies of 91.7% (normal), 93.3% (SCC), and 91.7% (BCC). We verified the potential use of collagen and lipid in the TME as biomarkers of tumor diversity and achieved accurate tumor classification based on the collagen and lipid content using PASA. The proposed method provides a new way to diagnose tumors.