Memristor, as a nonlinear element in nanometer size, is feasible to generate chaotic signals. Especially, it can improve the randomness of the signals and the complexity of chaotic systems. A novel multiscroll hyperchaotic system based on the flux-controlled memristor is designed. Its twin system with a different topological structure is obtained by varying only the flux variable of the memristor, which is a considerable difference from other chaotic systems existing in the literature. Scroll numbers of the proposed system and its twin, especially, are sensitive to the system parameters. To further investigate the characteristics of the system and its twin, their behaviors with respect to the strengths of the memristor, simulation time and coexistence of multiscroll hyperchaotic attractors are analyzed. Moreover, complex dynamical behaviors of the system and its twin are analyzed by phase diagrams, Poincaré cross-sections, Lyapunov indexes, power spectra, 0-1 test diagrams, and time series. Finally, a novel image encryption scheme is proposed based on the system and its twin. The system and its twin have independent key spaces, and the proposed algorithm makes them have an intersection, which greatly improves the key spaces. In addition, the security and reliability of the image encryption algorithm are demonstrated by different security analysis methods, including correlation analysis, robustness analysis and information entropy test. The results of all experiments prove that the proposed image encryption scheme is superior to other existing ones.