As the product mix in production changes dramatically, cell reconfiguration is requisite to smoothen the production process. Meanwhile, multiple production modes are simultaneously adopted in the site to promote productivity and assure flexibility, and thus the coordination scheduling among them becomes a challenging problem. To achieve cell reconfiguration and cell scheduling in a cellular manufacturing system in which no-idle flow-line and flexible job-shop production modes are hybridized, a mixed integer linear programming model is formulated and an enhanced adaptive multi-objective evolutionary algorithm is developed. In the proposed algorithm, a decision tree-based rule combination selector is developed to adaptively select the most appropriate rule combination fit for the given production scenario to generate a high-quality initial population. Three types of crossover operators and six objective-oriented local search operators are designed to increase the exploration and exploitation capability. An adaptive balance mechanism of exploration and exploitation is trained by Q-learning to maximize search efficiency. In addition, an adaptive adjustment mechanism of population size is designed to ensure diversity and speed up convergence. The comparative study demonstrates that the three proposed adaptive mechanisms are effective and the proposed algorithm with three adaptive mechanisms significantly outperforms other comparison algorithms in solving the studied problem.