Given an action of an affine algebraic group with only trivial characters on a factorial variety, we ask for categorical quotients. We characterize existence in the category of algebraic varieties. Moreover, allowing constructible sets as quotients, we obtain a more general existence result, which, for example, settles the case of a finitely generated algebra of invariants. As an application, we provide a combinatorial GIT-type construction of categorical quotients for actions of not necessarily reductive groups on, e.g. complete varieties with finitely generated Cox ring via lifting to the characteristic space.