Abstract This paper presents a novel and effective methodology for transmission expansion planning (TEP) in a double-auction based electricity market. The TEP problem has been formulated as a nonlinear programming problem to maximize the social welfare function of market participants (Generation and Distribution Companies), subject to power system operational and security constraints. The proposed methodology has been solved in two parts: in the first part, nonlinear TEP problem has been solved for a number of market scenarios to identify a set of prospective transmission lines; and in second part, multi-criteria data envelopment analysis has been applied on the identified set of prospective lines to select the most effective ones for transmission reinforcement. The stopping criterion of the TEP methodology has been applied based on the fact that increment in social welfare obtained by transmission reinforcement should be more than the investment cost of Transmission Company. The proposed methodology is well suited for nonlinear modeling of power systems as it reduces the complexity and computational burden of TEP problem substantially. To illustrate the efficacy of the proposed methodology, simulation result of Garver’s six-bus, IEEE 24-bus reliability test, and IEEE 662-bus systems have been presented.