Evaluating archaeobotanical data from over 3.9 million seeds and 124,300 charcoal fragments across 330 archaeological site phases in Southwest Asia, we reconstruct the history of olive and grape cultivation spanning a period of 6,000 years. Combining charcoal and seed data enables investigation into both the production and consumption of olive and grape. The earliest indication for olive and grape cultivation appears in the southern Levant around ca. 5000 BC and 4th millennium BC respectively, although cultivation may have been practiced prior to these dates. Olive and grape cultivation in Southwest Asia was regionally concentrated within the Levant until 600 BC, although there were periodic pushes to the East. Several indications for climate influencing the history of olive and grape cultivation were found, as well as a correlation between periods of high population density and high proportions of olive and grape remains in archaeological sites. While temporal uncertainty prevents a detailed understanding of the causal mechanisms behind these correlations, we suggest that long distance trade in olives, grapes and their associated products was integral to the economic, social, and demographic trajectories of the region.