The search of the new physics (NP) beyond the Standard Model is one of the most important topics in current high energy physics. With the increasing luminosities at the colliders, the search for NP signals requires the analysis of more and more data, and the efficiency in data processing becomes particularly important. As a machine learning algorithm, support vector machine (SVM) is expected to to be useful in the search of NP. Meanwhile, the quantum computing has the potential to offer huge advantages when dealing with large amounts of data, which suggests that quantum SVM (QSVM) is a potential tool in future phenomenological studies of the NP. How to use SVM and QSVM to optimize event selection strategies to search for NP signals are studied in this paper. Taking the tri-photon process at a muon collider as an example, it can be shown that the event selection strategies optimized by the SVM and QSVM are effective in the search of the dimension-8 operators contributing to the anomalous quartic gauge couplings.