Electric charging of colloidal particles in nonpolar solvents plays a crucial role for many industrial applications and products, including rubbers, engine oils, toners, or electronic displays. Although disfavored by the low solvent permittivity, particle charging can be induced by added surfactants, even nonionic ones, but the underlying mechanism is poorly understood, and neither the magnitude nor the sign of charge can generally be predicted from the particle and surfactant properties. The conclusiveness of scientific studies has been limited partly by a traditional focus on few surfactant types with many differences in their chemical structure and often poorly defined composition. Here we investigate the surface charging of poly(methyl methacrylate) particles dispersed in hexane-based solutions of three purified polyisobutylene succinimide polyamine surfactants with "subtle" structural variations. We precisely vary the surfactant chemistry by replacing only a single electronegative atom located at a fixed position within the polar headgroup. Electrophoresis reveals that these small differences between the surfactants lead to qualitatively different particle charging. In the respective particle-free surfactant solutions we also find potentially telling differences in the size of the surfactant aggregates (inverse micelles), the residual water content, and the electric solution conductivity as well as indications for a significant size difference between oppositely charged inverse micelles of the most hygroscopic surfactant. An analysis that accounts for the acid/base properties of all constituents suggests that the observed particle charging is better described by asymmetric adsorption of charged inverse micelles from the liquid bulk than by charge creation at the particle surface. Intramicellar acid-base interaction and intermicellar surfactant exchange help rationalize the formation of micellar ions pairs with size asymmetry.