Objective: Considering that some image encryption algorithms have the disadvantages of complex structure and high computational cost, and there are not many commonly used chaotic systems, which are easy to crack by attacks, to solve these problems, this paper proposes an image encryption algorithm based on three-dimensional nonlinear functions to solve these problems. Methods: The algorithm mainly combines the sinusoidal chaotic map with the ternary nonlinear function system to encrypt the image. Firstly, multiple ternary nonlinear function chaotic systems are designed. Then, the function iteration system is changed to invoke the computation of a specific expression under a random number; it is a chaotic sequence generated according to a chaotic mapping such as sine, and then the value of this chaotic sequence is used to select a ternary nonlinear function for iteration to obtain a chaotic sequence. Finally, the chaotic sequence performs the XOR and scrambling operations on the grey image Results: The algorithm has a simple structure, a better encryption effect, and more incredible difficulty deciphering. Moreover, through the phase diagram and bifurcation diagram, it can be seen that the system has good chaotic characteristics Conclusion: The method in this paper is novel; this method is a random variable order composite operation, which can not only be applied to image encryption but also can be used for fractal map generation and so on, and in some other chaotic fields will have a wide range of applications. It has essential research value.