In this paper, we introduce an invariant wavelet transform with multiple parameters in fractional domain and apply it to watermarking algorithm. First, we propose a new transform called redistributed invariant redundant fractional wavelet transform (RI-RFrWT) based on redistributed invariant integer wavelet transform (RI-IWT) and redundant fractional wavelet transform (RFrWT) which implements the extension of RI-IWT in the fractional domain. When the transform order of RI-RFrWT is 0, it degenerates to RI-IWT and has invariance. Second, RI-RFrWT is applied to the proposed optimized watermarking algorithm. This algorithm solves the problem that it is difficult for existing algorithms to show high robustness under both geometric and noise attacks. Third, a secure encryption algorithm based on Logisitic-adjusted-Sine map (LASM) and generalized Arnold transform (GAT) is designed to encrypt the information of watermark. Highly sensitive LASM and multi-parameter GAT improve the security of the watermarking algorithm. To improve the invisibility and robustness of the algorithm, particle swarm optimization (PSO) is also used to simultaneously improve the order of RI-RFrWT and embedding strength. Last, the simulation results demonstrate that the proposed watermarking method outperforms high invisibility, robustness and security. Compared with the relevant literature, it has obvious advantages in invisibility and resistance to geometric attacks and noise attacks.