(End Jo(N)) (0 Q of all endomorphisms of Jo(N) is equal to the algebra generated by the Hecke operators, regarded as Q-endomorphisms of J0(N). Our initial proof that this is indeed the case combined P. Deligne's theorem that Jo(N) is semi-stable over Q (in the sense of [6]) with a rather elaborate argument involving l-adic representations. This argument gave a proof that all endomorphisms of a semi-stable abelian variety over Q are rational whenever the variety has a certain endomorphism structure (similar to that of [4, Section 5]). With Shimura's help we were able to simplify the argument and at the same time make it more general. Eventually the principle emerged that all endomorphisms of a semi-stable variety are unramifted. (See (1.1) and (1.3) for precise statements.) Since Q has no unramified extensions, it follows that every endomorphism of a semi-stable abelian variety over Q is rational (i.e., defined over Q). We apply these results to Jo(N) in Section 3 and to certain other modular varieties in Section 4. The applications require an auxiliary result (based on an idea of W. Casselman) concerning the endomorphism algebra of an abelian variety with many real endomorphisms; the result is proved in Section 2.