Given the growing demand for simulating anisotropic material behavior at finite strains, constitutive modeling is in a challenging position to combine descriptive capabilities for several inelastic phenomena with the numerical feasibility for real-world applications. In this article, we develop a material model capable of reproducing anisotropy, viscoelasticity, stress softening, and permanent set by merging several pre-existing frameworks. Each constitutive effect is discussed separately in terms of its thermodynamics and mechanical interpretation and successively built on top of each other. Here, the pseudo-elastic approach to permanent set occupies a special place, with a novel discussion of its applicability to generic deformations. We show that the formulation does not lead to physical behavior in general, but can be constrained in such a way to produce appropriate stress predictions in an average sense. Examples of the stress response in several different deformation modes are visualized throughout. The capabilities and possible shortcomings of the formulation are highlighted and at the end a simple numerical algorithm for stress computation is presented.