The crystal-based primary collimators installed in the Large Hadron Collider (LHC) at CERN use the channelling process in bent crystals to steer halo particles efficiently onto downstream collimators. This scheme, called crystal collimation, is also considered for applications of fixed-target implementations in the context of the Physics Beyond Collider at the LHC. Crystal collimation uses 4 mm-long silicon crystals that need to be approached very close to the high-intensity circulating beams, posing obvious concerns for machine impedance. A complex mechanical assembly was developed for this purpose. The setup includes also a system to control with sub-μrad accuracy the angular orientation of the crystal, which is done with a high-precision interferometric system. In order to prevent possible beam-induced instabilities and/or damage of the device components from excessive RF-heating, the electromagnetic (EM) characterization of this device is essential prior to its usage with high-intensity beams. In this article, the longitudinal impedance of the crystal primary collimator is studied extensively and estimations of power loss inside the device are provided for the different beam types planned at the LHC and at its High-Luminosity upgrade (HL-LHC). Electromagnetic simulations are performed on a realistic model that includes all the relevant components. The model is described in detail and computational challenges coming from its complexity are discussed. Care is taken to characterize the materials of each relevant sub-component. In particular, the lossy properties of silicon, whose complex permittivity is also evaluated through RF rectangular-cavity perturbation measurements, are taken into account. Numerical results are then compared with dedicated RF measurements performed on a prototype built for the LHC.