Frankincense is used for analgesic, tumor-suppressive, and anti-inflammatory treatments in Traditional Chinese Medicine but poses toxicological concerns. Vinegar processing is a common technique used to reduce the toxicity of frankincense. This study aimed to investigate the chemical composition and quality evaluation of raw and vinegar-processing frankincense by multiple UPLC-MS/MS techniques. Additionally, we purposed refining the vinegar processing technique and identifying potentially harmful ingredients in the raw frankincense. Sub-chronic oral toxicity studies were conducted on raw and vinegar-processing frankincense in rats. The composition of frankincense was identified by UPLC-Q-TOF-MS/MS. Chemometrics were used to differentiate between raw and vinegar-processing frankincense. Potential chemical markers were identified by selecting differential components, which were further exactly determined by UPLC-QQQ-MS/MS. Moreover, the viability of the HepG2 cells of those components with reduced contents after vinegar processing was assessed. The toxicity of raw frankincense is attenuated by vinegar processing, among which vinegar-processing frankincense (R40) (herb weight: rice vinegar weight = 40:1) exhibited the lowest toxicity. A total of 83 components were identified from frankincense, including 40 triterpenoids, 37 diterpenoids, and 6 other types. The contents of six components decreased after vinegar-processing, with the lowest levels in R40. Three components, specifically 3α-acetoxy-11-keto-β-boswellic acid (AKBA), 3α-acetoxy-α-boswellic acid (α-ABA), and 3α-acetoxy-β-boswellic acid (β-ABA), inhibited the viability of HepG2 cells. The processing of frankincense with vinegar at a ratio of 40:1 could be an effective method of reducing the toxicity in raw frankincense. Our research improves understanding of the toxic substance basis and facilitates future assessments of frankincense quality.