The last decades have seen the consolidation of the cone-beam CT (CBCT) technology, which is nowadays widely used for different applications such as micro-CT for small animals, mammography, dentistry, or surgical procedures. Some CBCT systems may suffer mechanical strains due to the heavy load of the x-ray tube. This fact, together with tolerances in the manufacturing process, lead to different types of undesirable effects in the reconstructed image unless they are properly accounted for during the reconstruction. To obtain good quality images, it is necessary to have a complete characterization of the system geometry including the angular position of the gantry, the source-object and detector-object distances, and the position and pose of the detector. These parameters can be obtained through a calibration process done periodically, depending on the stability of the system geometry. To the best of our knowledge, there are no comprehensive works studying the effect of inaccuracies in the geometrical calibration of CBCT systems in a systematic and quantitative way. In this work, we describe the effects of detector misalignments (linear shifts, rotation, and inclinations) on the image and define their tolerance as the maximum error that keeps the image free from artifacts. We used simulations of four phantoms including systematic and random misalignments. Reconstructions of these data with and without errors were compared to identify the artifacts introduced in the reconstructed image and the tolerance to miscalibration deemed to provide acceptable image quality. Visual assessment provided an easy guideline to identify the sources of error by visual inspection of the artifactual images. Systematic errors result in blurring, shape distortion and/or reduction of the axial field of view while random errors produce streaks and blurring in all cases, with a tolerance which is more than twice that of systematic errors. The tolerance corresponding to errors in position of the detector along the tangential direction, that is, skew (<0.2°) and horizontal shift (<0.4mm), is tighter than the tolerance to those errors affecting the position along the longitudinal direction or the magnification, that is, vertical shift (<2mm), roll (<1.5°), tilt (<2°), and SDD (<3mm). We present a comprehensive study, based on realistic simulations, of the effects on the reconstructed image quality of errors in the geometrical characterization of a CBCT system and define their tolerance. These results could be used to guide the design of new systems, establishing the mechanical precision that must be achieved, and to help in the definition of an optimal geometrical calibration process. Also, the thorough visual assessment may be valuable to identify the most predominant sources of error based on the effects shown in the reconstructed image.