The notion of Schwarzian derivative for locally univalent holomorphic functions on complex plane was generalized for conformal diffeomorphisms by Osgood and Stowe in 1992 [27]. We shall identify a tensor that may serve as an analogue of the Schwarzian of Osgood and Stowe for CR mappings, and then use the tensor to define and study the CR M\"obius transformations and metrics of pseudo-hermitian manifolds. We shall establish basic properties of the CR Schwarzian and a local characterization of CR spherical manifolds in terms of fully integrability of the CR M\"obius equation. In another direction, we shall prove two rigidity results for M\"obius change of metrics on compact CR manifolds.