Abstract In the present study, we investigate the decays of the top quark $t\rightarrow c\gamma$, $t\rightarrow cg$, $t\rightarrow cZ$ and $t\rightarrow ch$. They are extremely rare processes in the Standard Model (SM). As the $U(1)$ extension of the minimal supersymmetric standard model (MSSM), the $U(1)_X$SSM features new superfields such as the right-handed neutrinos and three Higgs singlets. We analyze the effects of different sensitive parameters on the results and make reasonable theorecial predictions, which provides a useful reference for future experimental development. Considering the constraint of the updated experimental data, the numerical results show that the branching ratios of all four processes $t\rightarrow c\gamma,~cg,~cZ,~ch$ can reach the same order of magnitude as their experimental upper limits. Among them, $\tan\beta$ has the most obvious effect on each process and is the main parameter. $g_X$, $g_{YX}$, $\mu$, $M_2$, $\lambda_H$, $M_{U23}^2$ and $M_{Q23}^2$ are important parameters for the processes, and have effects on the numerical results. Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI. Article funded by SCOAP3 and published under licence by Chinese Physical Society and the Institute of High Energy Physics of the Chinese Academy of Science and the Institute of Modern Physics of the Chinese Academy of Sciences and IOP Publishing Ltd.