State-dependent queues are essential as they accurately capture the dynamic behavior of systems, enabling more realistic modeling and analysis of various real-world scenarios. The present investigation deals with a state-dependent single unreliable server finite queueing model under admission control F-policy. The admission control F-policy is a valuable tool to control the congestion problem of customers where the queue forms. In the proposed model, the server is prone to breakdown and can fail at any time. There is a two-phase repair facility available to address server failures. Chapman–Kolmogorov (C–K) steady-state equations based on the birth–death process are formed and solved using the matrix analytic method to formulate a mathematical model. By setting suitable values for state-dependent parameters, some particular models, such as the machine repair problem (MRP) and queueing model with customers’ balking and feedback, can also be deduced. Furthermore, various queueing indices are established to predict the validity of state-dependent queues. A total cost function for the system is constructed to enhance customer service while minimizing costs. This cost function is then optimized using a genetic algorithm (GA) and quasi-Newton method (QNM). The cost optimization in the state-dependent queueing model enables businesses to optimize resource allocation and achieve cost-efficient service delivery, leading to improved profitability and customer satisfaction.