The influence of solid-state drawing on the photo-actuation stress of oriented semi-crystalline polymers is explored experimentally and theoretically. Photo-responsive and ultra-drawn linear polyethylenes (PEs) containing a commercial benzotriazole dye (BZT) are studied as a model material. In accordance with previous studies, it is found that the actuation stress as a function of draw ratio exhibits an optimum and that the stress first increases and then decreases again slightly. A semi-empirical model is presented that describes the photo-actuation stress as a function of the draw ratio, Young's modulus, thermal expansion coefficient, and temperature difference between the illuminated and non-illuminated state. The Young's modulus and thermal expansion coefficient versus draw ratio curves are described using models based on the pseudo-affine deformation scheme in combination with a two-phase structural model in which highly oriented elements are connected in series with a disordered phase. Rather surprisingly, it is found that the model predicts that the photo-actuation stress is virtually independent of the draw ratio and the degree of chain orientation and extension. An excellent agreement is observed between model predictions and experimental results if the actuation stress is corrected for temperature effects related to the conversion of light into heat. The structure-property relationships derived here provide guidelines for the design of light-driven artificial muscle systems based on ultra-drawn polymers. For instance, it is predicted that ultra-drawn polyamides and polyesters are potentially excellent candidates for producing photo-actuators provided that suitable additives are used and this despite their low maximum attainable draw ratio, Young's modulus and strength.