We use a previously developed scattering-amplitudes-based framework for determining two-body Hamiltonians for generic binary systems with arbitrary spin S. By construction this formalism bypasses difficulties with unphysical singularities or higher-time derivatives. This framework has been previously used to obtain the exact velocity dependence of the O(G^{2}) quadratic-in-spin two-body Hamiltonian. We first evaluate the S^{3} scattering angle and two-body Hamiltonian at this order in G, including not only all operators corresponding to the usual worldline operators, but also an additional set due to an interesting subtlety. We then evaluate S^{4} and S^{5} contributions at O(G^{2}) which we confirm by comparing against aligned-spin results. We conjecture that a certain shift symmetry together with a constraint on the high-energy growth of the scattering amplitude specify the Wilson coefficients for the Kerr black hole to all orders in the spin and confirm that they reproduce the previously obtained results through S^{4}.