We present a Spitzer MIPS study of the decay of debris disk excesses at 24 and 70 $\mu$m for 255 stars of types F4 - K2. We have used multiple tests, including consistency between chromospheric and X-ray activity and placement on the HR diagram, to assign accurate stellar ages. Within this spectral type range, at 24 $\mu$m, $13.6 \pm 2.8 \%$ of the stars younger than 5 Gyr have excesses at the 3$\sigma$ level or more, while none of the older stars do, confirming previous work. At 70 $\mu$m, $22.5 \pm 3.6\%$ of the younger stars have excesses at $ \ge$ 3 $\sigma$ significance, while only $4.7^{+3.7}_{-2.2}$% of the older stars do. To characterize the far infrared behavior of debris disks more robustly, we double the sample by including stars from the DEBRIS and DUNES surveys. For the F4 - K4 stars in this combined sample, there is only a weak (statistically not significant) trend in the incidence of far infrared excess with spectral type (detected fractions of 21.9$^{+4.8}_{-4.3}\%$, late F; 16.5$^{+3.9}_{-3.3}\%$, G; and 16.9$^{+6.3}_{-5.0}\%$, early K). Taking this spectral type range together, there is a significant decline between 3 and 4.5 Gyr in the incidence of excesses with fractional luminosities just under $10^{-5}$. There is an indication that the timescale for decay of infrared excesses varies roughly inversely with the fractional luminosity. This behavior is consistent with theoretical expectations for passive evolution. However, more excesses are detected around the oldest stars than is expected from passive evolution, suggesting that there is late-phase dynamical activity around these stars.