Traditional power generation operation of reservoir mainly considers the maximization of power generation and guarantees the stability of the power system. However, blindly considering power generation objectives may ignore its impact on the ecological environment and navigation. A multi-objective optimization operation model considering power generation, ecological and navigation objectives is established in this paper. In order to efficiently solve the model, a new Improved Multi-objective Moth-flame Optimization Algorithm based on R-domination (R-IMOMFO) has been proposed. In order to enhance the ability of Moth-flame Optimization Algorithm (MFO) to overcome falling into the local optimum, it is improved from three aspects: update formula, inspiration of moth linear flight path and flame population update strategy, called improved MFO (IMFO) algorithm. In order to distinguish these individuals who are not dominated by each other in Pareto domination, the R-domination is proposed in combination with reference points. To verify the performance of IMFO and R-domination separately, different evolutionary algorithms and multi-objective mechanisms are combined to generate five new algorithms. Five new algorithms and five state-of-the-art algorithms are tested on the benchmark functions and reservoir operation model. The test results show that the proposed R-IMOMFO algorithm has the ability to obtain a set of solution with good convergence and strong distribution in the optimization operation problem of cascade reservoirs. Finally, the relationships between the objectives of the operation model are explored by the set of solution obtained by R-IMOMFO and the reason for the relationships is analyzed. The operation results show that the ecological demand and navigation demand have obvious contradictory relationships.