Extracellular vesicles (EVs) are nanosized intercellular messengers that bear enormous application potential as biological drug delivery vehicles. Much progress has been made for loading or decorating EVs with proteins, peptides or RNAs using genetically engineered donor cells, but post-isolation loading with synthetic drugs and using EVs from natural sources remains challenging. In particular, quantitative and unambiguous data assessing whether and how small molecules associate with EVs versus other components in the samples are still lacking. Here we describe the systematic and quantitative characterisation of passive EV loading with small molecules based on hydrophobic interactions - either through direct adsorption of hydrophobic compounds, or by membrane anchoring of hydrophilic ligands via cholesterol tags. As revealed by single vesicle imaging, both ligand types bind to CD63 positive EVs (exosomes), however also non-specifically to other vesicles, particles, and serum proteins. The hydrophobic compounds Curcumin and Terbinafine aggregate on EVs with no apparent saturation up to 106–107 molecules per vesicle as quantified by liquid chromatography – high resolution mass spectrometry (LC-HRMS). For both compounds, high density EV loading resulted in the formation of a population of large, electron-dense vesicles as detected by quantitative cryo-transmission electron microscopy (TEM), a reduced EV cell uptake and a toxic gain of function for Curcumin-EVs. In contrast, cholesterol tagging of a hydrophilic mdm2-targeted cyclic peptide saturated at densities of ca 104–105 molecules per vesicle, with lipidomics showing addition to, rather than replacement of endogenous cholesterol. Cholesterol anchored ligands did not change the EVs' size or morphology, and such EVs retained their cell uptake activity without inducing cell toxicity. However, the cholesterol-anchored ligands were rapidly shed from the vesicles in presence of serum. Based on these data, we conclude that (1) both methods allow loading of EVs with small molecules but are prone to unspecific compound binding or redistribution to other components if present in the sample, (2) cholesterol anchoring needs substantial optimization of formulation stability for in vivo applications, whereas (3) careful titration of loading densities is warranted when relying on hydrophobic interactions of EVs with hydrophobic compounds to mitigate changes in physicochemical properties, loss of EV function and potential cell toxicity.