Athermal systems across a large range of length scales, ranging from foams and granular bead packings to crumpled metallic sheets, exhibit slow stress relaxation when compressed. Experimentally they show a non-monotonic stress response when decompressed somewhat after an initial compression, i.e., under a two-step, Kovacs-like protocol. It turns out that from this response one can tell for how long the system was in a compressed state, suggesting an interpretation as a memory effect. In this paper we use a model of an athermal jammed solid, specifically a binary mixture of soft harmonic particles, to explore this phenomenon through in silico experiments. Using extensive simulations under conditions analogous to those in experiment, we observe identical phenomenology in the stress response under a two-step protocol. Our model system also recovers the behavior under a more recently studied three-step protocol, which consists of a compression followed by a decompression and then a final compression. We show that the observed response in both two-step and three-step protocols can be understood using linear response theory. In particular, a linear scaling with age for the two-step protocol arises generically for slow linear responses with power law or logarithmic decay and does not in itself point to any underlying aging dynamics.