This paper presents a new approach for solving a class of complicated nonlinear programming problems arises from optimal power flow with transient stability constraints (denoted by OTS) in power systems. By using a functional transformation technology proposed in Chen et al. (IEEE Trans. Circuits Syst. I Fundam. Theory Appl. 48:327---339, [2001]), the OTS problem is transformed to a semi-infinite programming (SIP). Then based on the KKT (Karush-Kuhn-Tucker) system of the reformulated SIP problem and the finite approximation technology, an iterative method is presented, which develops Wu-Li-Qi-Zhou' (Optim. Methods Softw. 20:629---643, [2005]) method. In order to save the computing cost, some typical computing technologies, such as active set strategy, quasi-Newton method for the subproblems coming from the finite approximation model, are addressed. The global convergence of the proposed algorithm is established. Numerical examples from power systems are tested. The computing results show the efficiency of the new approach.