Despite their enormous complexity and structural diversity, most biological materials show a remarkably similar viscoelastic phenomenology: nonlinear elasticity, power-law or logarithmic stress relaxation, and plastic length adaptation. Here we present a simple model based on Huxley's sliding filament model to demonstrate that such behavior can arise from generic structural properties, independent of the actual molecular constituents of the system. We compare the model predictions to data from active and passive microrheological experiments on epithelial cells and fibroblasts, smooth muscle tissue, and extracellular matrix protein networks.The material is represented by an uniaxial arrangement of infinitely stiff filaments crosslinked with parallel elastic elements that have a distribution of attachment angles. When the system is sheared or stretched, elements start to align, leading to strain stiffening due to a geometric recruitment of springs. The elastic elements have force-dependent average lifetimes described by energy traps with a broad distribution of energy trap depths. Broken links can reattach at random positions and attachment angles after unbinding. Such nanoscale structural rearrangements lead to viscous flow and plastic length adaptation on a macroscopic scale. Due to a broad distribution of energy trap depths, the system displays power law stress relaxation and soft glassy rheology.The model is capable of qualitatively reproducing experiments, and gives quantitative agreement for creep compliance, stress stiffening and plasticity in the case of cell microrheology. These results suggest that recruitment and dynamic unbinding of elastic elements are the common mechanism underlying the mechanical behavior of many complex biological materials from single cells to whole tissues.