In this paper, a fractional-order hyperchaotic detuned laser system (FHDLS) is proposed, and a novel image cryptosystem with high-security performance is designed by combining FHDLS, improved shuffling algorithm, and DNA mutation diffusion algorithm. Firstly, the complex dynamics of FHDLS is researched by phase diagram, bifurcation diagram, Lyapunov exponential spectrum, and spectral entropy (SE) complexity. The attractor coexistence phenomenon of the system is analyzed from different initial values for given parameters. Meanwhile, the randomness of the proposed FHDLS is verified by the correlation. Then, the FHDLS is implemented by the analog circuit and the digital hardware (Digital Signal Processing, DSP), respectively. Finally, make use of these excellent characteristics, and perturb the initial values and chaotic state of the system through Secure Hash Algorithm 256 (SHA-256). Then the pixel-level and bit-level replacement and transformation operations are performed on the image respectively, to implement a chaos-based image cryptosystem. The regimen performs one scrambled and diffusion together and one diffusion operation to achieve maximum confusion and diffusion. To demonstrate the safety performance of the proposed image cryptographic regimen, standard security analyses is adopted, including histogram, correlation, information entropy, security key, anti-differential attack, robustness, etc., and it is compared with the existing advanced algorithms. The experimental results indicate that the regimen works well in encryption and can resist various attacks effectively, and the FHDLS has wide application prospects in image encryption.