Abstract Recent timing observation provides an intriguing result for the braking index of the X-ray pulsar PSR$\:$J1640$-$4631, which has a measured braking index $n=3.15\pm 0.03$. The decrease of the inclination angle between the spin axis and the magnetic axis can be responsible for such a high braking index. However, the physical mechanisms causing the change of the magnetic inclination angle have not been fully understood. In this Letter, we apply a two-dipole model given by Hamil, Stone, and Stone (2016, Phys. Rev. D., 94, 063012) to explain the decrease of the magnetic inclination angle of PSR$\:$J1640$-$4631. The rotation effect of a charged sphere and the magnetization of ferromagnetically ordered material produce magnetic moments $M_{\,1}$ and $M_{\,2}$, respectively. There exists a minimum of the potential energy for the magnetic moment $M_{\,2}$ in the magnetic field of $M_{\,1}$, hence the $M_{\,2}$ will freely rotate around the minimum energy position (i.e., equilibrium position), similar to a simple pendulum. Our calculation indicates that the magnetic moment $M_{\,2}$ would evolve towards alignment with the spin axis for PSR$\:$J1640$-$4631, and cause the magnetic inclination angle to decrease. The single peak in the pulse profile favors a relatively low change rate for the magnetic inclination angle.