Secrete permutations are widely used in multimedia encryption algorithms, and many permutation-only ciphers have been proposed in recent years for protection of digital images and videos. Recently, ergodic matrices are used to realize the pureposition permutation algorithms. An effective decryption algorithm is developed for all pureposition permutation algorithms. Thereby, it is pointed out that all permutation only image ciphers are insecure against known/chosen-plaintext attacks. This paper presents an image encryption scheme that uses an adaptive ergodic matrix for each block encryption (by adapting the ergodic matrix from one block to another), and combines shuffling the positions, by adaptive ergodic matrix, and changing the grey values of image pixels based on preprocessing the discrete output signal of the Chen’s chaotic system. Thereby, significantly increasing the resistance to known/chosen-plaintext and statistical attacks. Moreover, visually and computationally, experimental results demonstrate the high security and significantly superior encryption quality of the proposed scheme.