This paper presents the high-fidelity physical modeling of a Thrust Vector Control (TVC) system operated by Electro-Mechanical Actuators (EMAs) for Reusable Launch Vehicles (RLVs). In contrast to simplified, often linear, models, high-fidelity physical models enable a better assessment of the actual system performance and a deeper verification of the on-board software robustness against disturbances, unmodeled dynamics or degradation. This aspect is essential to enable the reusability concept as driven by long-term reliability requirements. Moreover, critical systems like Fault Detection, Isolation and Recovery (FDIR) logic, whose robustness and efficacy are often difficult to prove, can be thoroughly tested without requiring hardware experiments with eventual invasive modifications for fault injections. This work captures the whole dynamics of the EMA and TVC components, including the power drive electronics, the electrical motor, and the mechanical transmission for the EMA, as well as the mechanical properties of the engine nozzle acting as a load. The resulting differential algebraic equation system is modeled in the Modelica acausal object-oriented modeling language. An ad-hoc framework is implemented to guarantee flexibility and modularity, which allows models with different fidelity levels to be easily exchanged to achieve the simulation objectives and needed accuracy level. The impact of the different models on the closed-loop TVC dynamics is analyzed in both the frequency and time domain, and then benchmarked with a realistic RLV mission. The results show that the high-fidelity physical models provide a better understanding of the more complex effects governing the TVC dynamics and can, in turn, effectively improve its requirement definition process.