The accurate evaluation and fair comparison of wind farms power generation performance is of great significance to the technical transformation and operation and maintenance management of wind farms. However, problems exist in the evaluation indicator systems such as confusion, coupling and broadness, and the influence of wind energy resource differences not being able to be effectively eliminated, which makes it difficult to achieve the fair comparison of power generation performance among different wind farms. Thus, the evaluation indicator system and comprehensive evaluation method of wind farm power generation performance, including the influence of wind energy resource differences, are proposed in this paper to address the problems above, to which some new concepts such as resource conditions, ideal performance, reachable performance, actual performance, and performance loss are introduced in the proposed indicator system; the combination of statistical and comparative indicators are adopted to realize the quantitative evaluation, indicator decoupling, fair comparison, and loss attribution of wind farm power generation performance. The proposed comprehensive evaluation method is based on improved CRITIC (Criteria Importance though Intercrieria Correlation) weighting method, in which the uneven situation of different evaluation indicators and the comprehensive comparison of power generation performance among different wind farms shall be overcome and realized. Several sets of data from Chinese wind farms in service are used to validate the effectiveness and applicability of the proposed method by taking the comprehensive evaluation models based on CRITIC weighting method and entropy weighting method as the benchmarks. The results demonstrated that the proposed evaluation indicator system works in the quantitative evaluation and fair comparison of wind farm design, operation, and maintenance and traces the source of power generation performance loss. In addition, the results of the proposed comprehensive evaluation model are more in line with the actual power generation performance of wind farms and can be applied to the comprehensive evaluation and comparison of power generation performance of different wind farms.