By coupling memristors to nonlinear circuits, more complex dynamical behaviors can be induced. However, to date, there has been insufficient attention given to high-dimensional chaotic systems based on memristors. In this paper, a magnetic-controlled memristor is combined with a three-dimensional chaotic system, resulting in a five-dimensional memristive chaotic system. Through dynamic analysis and numerical simulations, the chaotic nature of the system is elucidated based on fundamental system behaviors, including Lyapunov dimension, dissipativity, stability of equilibrium points, 0–1 test, and Poincaré mapping. During the complex dynamical analysis of this system, unique dynamical behaviors are discovered, including intermittent chaos, transient chaos, extreme multistability, and offset-boosting. Moreover, the consistency between numerical calculations and the physical implementation of the actual system is verified through equivalent circuit design. Finally, this system is applied to image encryption, leading to the design of an efficient and secure hyper-chaotic image encryption algorithm, whose effectiveness is confirmed through several security tests.