The physical mechanisms that link the termination of star formation in quiescent galaxies and the evolution of their baryonic components, stars, and the interstellar medium (ISM; dust, gas, and metals) are poorly constrained beyond the local Universe. In this work, we characterise the evolution of the dust content in 545 quiescent galaxies observed at 0.1 < z < 0.6 as part of the hCOSMOS spectroscopic redshift survey. This is, to date, the largest sample of quiescent galaxies at intermediate redshifts for which the dust, stellar, and metal abundances are consistently estimated. We analyse how the crucial markers of a galaxy dust life cycle, such as specific dust mass (Mdust/M⋆), evolve with different physical parameters, namely gas-phase metallicity (Zgas), time since quenching (tquench), stellar mass (M⋆), and stellar population age. We find morphology to be an important factor in the large scatter in Mdust/M⋆ (∼2 orders of magnitude). Quiescent spirals exhibit strong evolutionary trends of specific dust mass with M⋆, stellar age, and galaxy size, in contrast to the little to no evolution experienced by ellipticals. When transitioning from solar to super-solar metallicities (8.7 ≲ 12 + log(O/H)≲9.1), quiescent spirals undergo a reversal in Mdust/M⋆, indicative of a change in dust production efficiency. By modelling the star formation histories of our objects, we unveil a broad dynamical range of post-quenching timescales (60 Myr < tquench < 3.2 Gyr). We show that Mdust/M⋆ is highest in recently quenched systems (tquench < 500 Myr), but its further evolution is non-monotonic, as a consequence of different pathways for dust formation, growth, or removal on various timescales. Our data are best described by simulations that include dust growth in the ISM. While this process is prevalent in the majority of galaxies, for ∼15% of objects we find evidence of additional dust content acquired externally, most likely via minor mergers. Altogether, our results strongly suggest that prolonged dust production on a timescale of 0.5 − 1 Gyr since quenching may be common in dusty quiescent galaxies at intermediate redshifts, even if their gas reservoirs are heavily exhausted (i.e. cold gas fraction < 1 − 5%).