Abstract Recent research on non-Abelian phenomena has cast a new perspective on controlling light. In this work, we provide a simple and general approach to induce non-Abelian gauge field to tremble the light beam trajectory. With in-plane duality symmetry relaxed, our theoretical analysis finds that non-Abelian electric field can be synthesized through a simple real-space rotation of any biaxial material. With orthogonal optical modes excited, their interference leads to an oscillation of the propagating optical beam, which is a direct consequence of the emergence of non-Abelian electric field, influencing light in a manner similar with how electric fields act on charged particles. Our microwave experiments provide unambiguous evidence to the observation of such an optical Zitterbewegung effect where excellent agreement can be found between theorical derivation, numerical simulations and experiments. By extending the idea to optical regime using natural material, we here provide another example to shake the general intuition that light travels in straight lines in homogeneous media.