If a helical network of fibers is embedded in a swellable matrix, and if the fibers themselves resist swelling, then a change in the amount of swelling agent will cause a corresponding twisting motion in the material. This effect has recently been analyzed in highly deformable soft material tubes using the theory of hyperelasticity, suitably modified to incorporate the swelling effect. Those studies examined the effect of spiral angle and fiber-to-matrix inherent stiffness in the context of a ground state matrix material that exhibited classical neo-Hookean behavior in the absence of swelling. While such a ground state material is nonlinear in general, its shear response is linear. As we describe here, it is this shear response that governs the matrix contribution to the twist-swelling interaction. Because gels, elastomers, and even biological tissue can exhibit complex ground state behavior in shear—behavior that may depart significantly from a linear response—we then examine the effect of alternative ground state behaviors on the twist-swelling interaction. The range of behaviors considered includes materials that harden in shear, materials that soften in shear, materials that have an ultimate shear stress bound, and materials that collapse in shear. Matrix materials that either soften or collapse in shear are found to amplify the twisting effect.