The formation of fibrillar aggregates has long been associated with neurodegenerative disorders such as Alzheimer and Parkinson diseases. Although fibrils are still considered important to the pathology of these disorders, it is now widely understood that smaller amyloid oligomers are the toxic entities along the misfolding pathway. One characteristic shared by the majority of amyloid oligomers is the ability to disrupt membranes, a commonality proposed to be responsible for their toxicity, although the mechanisms linking this to cell death are poorly understood. Here, we describe the physical basis for the cytotoxicity of oligomers formed by the prion protein (PrP)-derived amyloid peptide PrP(106-126). We show that oligomers of this peptide kill several mammalian cells lines, as well as mouse cerebellar organotypic cultures, and we also show that they exhibit antimicrobial activity. Physical perturbation of model membranes mimicking bacterial or mammalian cells was investigated using atomic force microscopy, polarized total internal reflection fluorescence microscopy, and NMR spectroscopy. Disruption of anionic membranes proceeds through a carpet or detergent model as proposed for other antimicrobial peptides. By contrast, when added to zwitterionic membranes containing cholesterol-rich ordered domains, PrP(106-126) oligomers induce a loss of domain separation and decreased membrane disorder. Loss of raft-like domains may lead to activation of apoptotic pathways, resulting in cell death. This work sheds new light on the physical mechanisms of amyloid cytotoxicity and is the first to clearly show membrane type-specific modes of action for a cytotoxic peptide.