Single-pixel imaging (SPI) can transform 2D or 3D image data into 1D light signals, which offers promising prospects for image compression and transmission. However, during data communication these light signals in public channels will easily draw the attention of eavesdroppers. Here, we introduce an efficient encryption method for SPI data transmission that uses the 3D Arnold transformation to directly disrupt 1D single-pixel light signals and utilizes the elliptic curve encryption algorithm for key transmission. This encryption scheme immediately employs Hadamard patterns to illuminate the scene and then utilizes the 3D Arnold transformation to permutate the 1D light signal of single-pixel detection. Then the transformation parameters serve as the secret key, while the security of key exchange is guaranteed by an elliptic curve-based key exchange mechanism. Compared with existing encryption schemes, both computer simulations and optical experiments have been conducted to demonstrate that the proposed technique not only enhances the security of encryption but also eliminates the need for complicated pattern scrambling rules. Additionally, this approach solves the problem of secure key transmission, thus ensuring the security of information and the quality of the decrypted images.