Hafnium oxide is widely used in optical coating materials because of its high refractive index, broad band gap, high laser damage threshold and its high transparency in the wavelength range from ultraviolet to middle infrared (0.20–12 μm). However, hafnium oxide has some defects such as strong crystallinity and poor surface quality, which results in enhanced absorption and reflection of light wave and limits its application in optical field. As a pentoxide with high transparency and excellent optical properties in the wavelength range from ultraviolet to mid-infrared(0.35–9 μm), the doping of niobium pentoxide can effectively improve the surface quality and optical properties of the films. Mixed films with different chemical compositions were prepared by ion beam assisted deposition, and the microstructure, crystallinity, surface quality, optical properties and mechanical properties of the mixed films with different chemical compositions were investigated. The doping of niobium pentoxide inhibits the crystallization of hafnium oxide, optimizes the surface quality of the films, and improves the refractive index and mechanical properties of mixed films, effectively broadens the application of hafnium oxide in the field of optical films.