Dielectric elastomer (DE) transducers are known to exhibit a rate-dependent hysteresis in their force-displacement response, which is commonly attributed to the viscoelastic behavior of elastomer materials and compliant electrodes. In the case of DE materials characterized by low mechanical losses, such as silicone, the mechanical hysteresis often turns out to be practically rate-independent in the low frequency range (sub-Hz), whereas rate-dependent hysteretic effects only become relevant at higher deformation rates. Most of the existing literature focuses on describing DE hysteretic losses using viscoelasticity theory. This approach results in relatively simple dynamic models, which are not capable of describing rate-independent hysteretic behaviors. In this work, we propose a control-oriented modeling framework for both rate-dependent and rate-dependent hysteresis occurring in uniaxially loaded DE actuators. To this end, classic thermodinamically-consistent modeling approaches for DEs are combined with a new energy-based Maxwell-Lion formalization of the hysteretic losses. The resulting dynamic model comprises a set of nonlinear ordinary differential equations, and is capable of simultaneously describe geometric dependencies, large deformation nonlinearities, electro-mechanical coupling, and rate-independent and rate-dependent hysteretic effects. To deal with the large number of involved parameters, a novel systematic identification algorithm based on quadratic programming is also proposed. After presenting the theory, the model is validated based on experiments conducted on a silicone-based rolled DE actuator. Its superiority compared to classic DE viscoelastic models is quantitatively assessed.