There is an ongoing interest in the research and application of thin film coatings containing titanium oxides and titanium nitrides, due to their properties such as photocatalytic activity, mechanical hardness, biocompatibility, and so on. Reactive sputter deposition is widely used for the production of such coatings. A characteristic of this method is the hysteresis, which leads to an ambiguous dependence of the deposition rate and the coating quality on the process conditions. There are a number of theoretical models describing reactive magnetron sputtering. One of the most advanced is the RSD2013 model. Several parameters in this model can only be determined experimentally. This article focuses on the investigation of the hysteresis during the reactive magnetron sputtering deposition process. The RSD2013 parameters that describe the experimental setup were determined, and on their basis the dependences of the characteristics of the hysteresis region on the discharge power, the type of reactive gas, and the working gas pressure were investigated. Additionally, the growth rate of the thin film coating prepared in various modes was compared with the RSD2013 model, which is in agreement with the experimental data. The results obtained from the RSD2013 model in comparison to the experimentally obtained data make it possible to provide an overview of suitable operating modes for the deposition of titanium dioxide and titanium dioxide for the magnetron sputtering device used.