Recently, variational methods were successfully applied for computing the optical flow in gray and RGB-valued image sequences. A crucial assumption in these models is that pixel-values do not change under transformations. Nowadays, modern image acquisition techniques such as electron backscatter tomography (EBSD), which is used in material sciences, can capture images with values in nonlinear spaces. Here, the image values belong to the quotient space $\text{SO}(3)/ \mathcal S$ of the special orthogonal group modulo the discrete symmetry group of the crystal. For such data, the assumption that pixel-values remain unchanged under transformations appears to be no longer valid. Hence, we propose a variational model for determining the optical flow in $\text{SO}(3)/\mathcal S$-valued image sequences, taking the dependence of pixel-values on the transformation into account. More precisely, the data is transformed according to the rotation part in the polar decomposition of the Jacobian of the transformation. To model non-smooth transformations without obtaining so-called staircasing effects, we propose to use a total generalized variation like prior. Then, we prove existence of a minimizer for our model and explain how it can be discretized and minimized by a primal-dual algorithm. Numerical examples illustrate the performance of our method.