ApoE isoforms are major genetic risk factors related to Alzheimer's disease (AD), which is associated with the aggregation of the amyloid beta (Aβ) peptide. However, a molecular-level understanding of how ApoE is involved in AD pathology is still lacking. Here we employ a fluorescent lifetime-based assay that reports a specific ApoE-dependent modification of Aβ, both in vitro and in neuronal cells. We explore this phenomenon in several types of cells where the change of lifetime of Aβ oligomers correlates with the relative ApoE content, with the highest effect displayed by primary-cultured rat astrocytes. At the single-molecule level, ApoE-induced modification also produces distinct Aβ oligomers with increased affinity to artificial lipid bilayers. A known peptide inhibitor of Aβ-ApoE interaction, LVFFA, diminishes both the lifetime modification and toxicity of Aβ oligomers in a dose-dependent manner. The correlation between the short lifetime component and Aβ toxicity was corroborated by experiments using two nontoxic Aβ variants. Our data, therefore, establish a clear correlation between the lifetime modification of Aβ and its toxicity. Finally, neuronal stem cells derived from cells obtained from Alzheimer's patients also show the short-lifetime component, the changed oligomeric state, and the increased membrane affinity, which is likely to be relevant for AD. In summary, our assay can provide a direct readout of the toxic ApoE-Aβ interaction in vitro and in cells, which can provide a platform for AD drug discovery.