With the increasing reliance on cloud computing as the foundational infrastructure for technological operations, the urgency and efficacy with which failed virtual machines can be successfully repaired earns significant weight. In this study, we have presented a queueing model that analyzes the repair process of failed virtual machines in cloud computing system. This model includes various important components, such as control N-policy, server unreliability, discouragement behavior of failed virtual machines (like balking and reneging), retention of reneged machines, redundancy, and two-level Bernoulli feedback. A steady-state performance analysis has been performed. The proposed model provides practical suggestions for cloud service providers to improve the reliability and availability of their services, which insights more than theoretical frameworks. This study is unique in the field of queueing models of cloud computing machining environment, as it incorporates parallel consideration of control N policies, balking, reneging, retention, server vacation, two-level Bernoulli feedback and unreliable server behavior. A thorough examination of the behavior of key performance indicators has been conducted and presented using detailed graphical representations. In addition, a cost function is formulated for the proposed model. Further, the Grey Wolf Optimization (GWO) and Particle Swarm Optimization (PSO) techniques are utilized to optimize the total cost function, thereby gaining valuable insights into the system’s behavior across various conditions as well as to compare the outcomes of GWO and PSO.