Over the last two decades, considerable research has built on remote sensing of spectral diversity to assess plant diversity. The spectral variation hypothesis (SVH) proposes that spatial variation in reflectance data of an area is positively associated with plant diversity. While the SVH has exhibited validity in dense forests, it performs poorly in highly fragmented and temporally dynamic agricultural landscapes covered mainly by grasslands. Such underperformance can be attributed to the mosaic-like spatial structure of human-dominated landscapes with fields in varying phenological and management stages. Therefore, we argued for re-evaluating SVH’s flawed window-based spatial analysis and underutilized temporal component. In particular, we captured the spatial and temporal variation in reflectance and assessed the relationships between spatial and temporal components of spectral diversity and plant diversity at the parcel level as a unit that relates to management patterns. Our investigation spanned three grasslands on two continents covering a wide spectrum of agricultural usage intensities. To calculate different components of spectral diversity, we used multi-temporal spaceborne Sentinel-2 data. We showed that plant diversity was negatively associated with the temporal component of spectral diversity across all sites. In contrast, the spatial component of spectral diversity was related to plant diversity in sites with larger parcels. Our findings highlighted that in agricultural landscapes, the temporal component of spectral diversity drives the spectral diversity-plant diversity associations. Consequently, our results offer a novel perspective for remote sensing of plant diversity globally.