Internet communication is one of the significant aspects of modern civilization. People use banking, health, social, or shopping platforms and send a lot of data. Each communication should be secured and protected against dishonest users’ activities during its transfer via network links. Cryptographic protocols provide such security and protection. Because of the evolution of the vulnerabilities and attackers’ methods, the cryptographic protocols should be regularly verified. This paper presents a methodology for testing the security of cryptographic protocols using the CMMTree framework. We developed and adapted a software package for analyzing cryptographic protocols regarding compatibility with the CMMTree framework using a predicate according to the approach described in Siedlecka-Lamch et al.’s works. We optimized and strengthened the mentioned approach with tree optimization methods and a lexicographic sort rule. Next, we researched the well-known security protocols using a developed tool and compared and verified the results using sorted and shuffled data. This work produced promising results. No attacks on the tested protocols were discovered.