A new fractional-order 5D hyperchaotic system based on memristor is constructed in this paper, with the speciality that the system exists chaotic and hyperchaotic states in the fractional-order case, while in periodic state in the integer-order. In addition, it has a variety of special phenomena at fractional-order such as infinite initial value range, parameter-dependent offset-boosting and amplitude control, attractor coexistence, and fractional order complexity greater than integer order. The correctness and feasibility of the system is verified by analog circuit simulation and hardware circuit implementation. Combining this system with image encryption algorithms, two new scrambling algorithms and a diffusion algorithm are proposed. And a high-quality encryption scheme that can be applied to a wide range of color images is proposed. The scheme is found to have excellent security after verification by various security analyses and comparison with other literatures. This paper provides a basis for the superiority of fractional-order chaotic systems and provides new methods in the field of image encryption.