Mass spectrometry (MS) is a powerful technology for the structural elucidation of known or unknown small molecules. However, the accuracy of MS-based structure annotation is still limited due to the presence of numerous isomers in complex matrices. There are still challenges in automatically interpreting the fine structure of molecules, such as the types and positions of substituents (substituent modes, SMs) in the structure. In this study, we employed flavones, flavonols, and isoflavones as examples to develop an automated annotation method for identifying the SMs on the parent molecular skeleton based on a characteristic MS/MS fragment ion library. Importantly, user-friendly software AnnoSM was built for the convenience of researchers with limited computational backgrounds. It achieved 76.87% top-1 accuracy on the 148 authentic standards. Among them, 22 sets of flavonoid isomers were successfully differentiated. Moreover, the developed method was successfully applied to complex matrices. One such example is the extract of Ginkgo biloba L. (EGB), in which 331 possible flavonoids with SM candidates were annotated. Among them, 23 flavonoids were verified by authentic standards. The correct SMs of 13 flavonoids were ranked first on the candidate list. In the future, this software can also be extrapolated to other classes of compounds.