The self-interaction error (SIE), i.e. unphysical interactions of electrons with themselves, has plagued developers and users of Density Functional Approximations (DFAs) since the inception of Density Functional Theory (DFT). Formally, it can be separated into the one-electron and many-electron SIE; herein we present one of the most comprehensive studies of the first. While we focus mostly on the total SIE, we also make use of two different decompositions. The first is a separation into functional and density-driven errors as championed by Sim, Burke and co-workers [J. Phys. Chem. Lett., 2018, 9, 6385-6392]; the second separates the error into exchange, correlation, and one-electron components, with the latter being a density error that has not been discussed in this form before. After investigating the familiar hydrogen atom and dihydrogen cation, we establish a relationship between the SIE and the nuclear charge with the help of a series of heavier hydrogenic analogues. For the mononuclear systems and the diatomics at the dissociation limit, this relationship is linear in nature with prominent exceptions, mostly belonging to the Minnesota and range-separated (double-)hybrid DFAs. For the first time, we also show how the magnitude of the SIE depends on the underlying atomic-orbital basis set and how DFAs that rely on a popular van-der-Waals DFT type London-dispersion term exhibit "self-dispersion". We find that range separation is not a panacea for solving the one-electron SIE. DFAs that have been developed to be one-electron SIE free for one system, such as the hydrogen atom, show larger errors for heavier hydrogenic systems. Often, one-electron SIE-free DFAs rely on fortuitous error cancellation between their exchange and correlation components. An analysis of the most robust methods for general applications to date reveals that they suffer moderately from the one-electron SIE, while DFAs that are nearly SIE-free do not perform well in applications. Implicit in the continued existence of the one-electron SIE is that well-performing DFAs continue to suffer insufficiencies at their fundamental levels that are being compensated for by the SIE. Our analysis includes more than 250 000 datapoints, resulting in multiple insights that may drive future developments of new DFAs or SIE corrections.