Modular categories are a well-known source of quantum 3-manifold invariants. In this paper we study structures on modular categories which allow to define refinements of quantum 3-manifold invariants involving cohomology classes or generalized spin and complex spin structures. A crucial role in our construction is played by objects which are invertible under tensor product. All known examples of cohomological or spin type refinements of the Witten-Reshetikhin-Turaev 3-manifold invariants are special cases of our construction. In addition, we establish a splitting formula for the refined invariants, generalizing the well-known product decomposition of quantum invariants into projective ones and those determined by the linking matrix.