The AKS algorithm is an important breakthrough in showing that primality testing of an integer can be done in polynomial time. In this paper, we study the optimization of its runtime. Namely, given a finite cardinality set of alphabets of a deterministic polynomial runtime Turing machine and the number of strings of an arbitrary input integer whose primality is to be tested as the system parameters, we consider the randomized AKS primality testing function as the objective function. Under randomization of the system parameters, we have shown that there are definite signatures of the local and global instabilities in the AKS algorithm. We observe that instabilities occur at the extreme limits of the parameters. It is worth mentioning that Fermat’s little theorem and Chinese remaindering help with the determination of the underlying stability domains. On the other hand, in the realm of the randomization theory, our study offers fluctuation theory structures of the AKS primality testing of an integer through its maximum number of irreducible factors. Finally, our optimization theory analysis anticipates a class of real-world applications for future research and developments, including optimal online security, system optimization and its performance improvements, (de)randomization techniques, and beyond, e.g., polynomial time primality testing, identity testing, machine learning, scientific computing, coding theory, and other stimulating optimization problems in a random environment.