The main purpose of this work is to develop a three-dimensional (3D) viscoelastic rod model for hard-magnetic soft (HMS) rods under large deformation which are widely used active structures in soft robotics. To do so, the Simo’s viscoelasticity theory has been rationally incorporated into the geometrically exact 3D curved rod model. The proposed model includes the deformation modes of axial tension, shear, bending, and torsion, which is applicable to the HMS rods with arbitrarily initial curved and twisted geometries under 3D large deformation. The viscoelastic constitutive equations of the HMS rod in the present formulation are formulated, which include the general relaxation functions. To obtain the expression for the magnetic load, the rotation-based magnetic free energy density is introduced, and the governing equations of the HMS rod with magnetic load and body force are presented. To obtain the numerical implementation, an implicit time integration algorithm that simply extends the generalized-α method for the rotation group, and the corresponding variational formulation and its linearization of the rod model are derived. To validate the model, five numerical examples, including 2D dynamic buckling, 3D static, and 3D dynamic problem are presented. The dynamic problems include the dynamic snap-through behavior of a bistable HMS arch and damped oscillation of a quarter arc cantilever under 3D deformation. The simulation results show good agreement with the results reported in the literature.