Utilizing the adopted average topographic density of 2670 kg/m3 in the reduction of gravity anomalies introduces errors attributed to topographic density variations, which consequently affect geoid modeling accuracy. Furthermore, the mean gravity along the plumbline within the topography in the definition of Helmert orthometric heights is computed approximately by applying the Poincaré-Prey gravity reduction where the topographic density variations are disregarded. The Helmert orthometric heights of benchmarks are then affected by errors. These errors could be random or systematic depending on the specific geological setting of the region where the leveling network is physically established and/or the geoid model is determined. An example of systematic errors in orthometric heights can be given for large regions characterized by sediment or volcanic deposits, the density of which is substantially lower than the adopted topographic density used in Helmert's definition of heights. The same applies to geoid modeling errors. In this study, we investigate these errors in the Hong Kong territory, where topographic density is about 20% lower than the density of 2670 kg/m3. We use the digital rock density model to estimate the effect of topographic density variations on the geoid and orthometric heights. Our results show that this effect on the geoid and Helmert orthometric heights reach maxima of about 2.1 and 0.5 cm, respectively. Both results provide clear evidence that rock density models are essential in physical geodesy applications involving gravimetric geoid modeling and orthometric height determination despite some criticism that could be raised regarding the reliability of these density models. However, in regions dominated by sedimentary and igneous rocks, the geological information is essential in these applications because topographic densities are substantially lower than the average density of 2670 kg/m3, thus introducing large systematic errors in geoid and orthometric heights.