The impact of the optical anisotropy of the cavity material on the cavity-photon dispersion was investigated for one-dimensional (1D) confined microresonators. This is relevant for birefringent cavity materials such as GaN or ZnO. An analytical expression for the dispersion of the cavity-photon mode in energy and broadening was derived. In the general case, two cavity-photon modes appear in each of the TE- and TM-polarized spectra. These two modes correspond to the two electromagnetic waves which propagate within the cavity medium having the same energy however differing in their propagation direction and velocity. We show that the anisotropy has a strong impact on the TE-TM splitting. Even for the case where the optical axis is oriented perpendicularly to the layer interface the influence of the anisotropy on the TE-TM splitting is in the same order of magnitude as the splitting itself. Therefore, optical anisotropy of the cavity cannot be neglected for a precise description of the cavity-photon mode energy. We applied this model to an all-oxide microresonator with a ZnO cavity and found a good agreement with the experimentally observed dispersion of the cavity-photon modes.