The quantum electrodynamical (QED) process of Compton scattering in strong magnetic fields is commonly invoked in atmospheric and inner magnetospheric models of x-ray and soft gamma-ray emission in high-field pulsars and magnetars. A major influence of the field is to introduce resonances at the cyclotron frequency and its harmonics, where the incoming photon accesses thresholds for the creation of virtual electrons or positrons in intermediate states with excited Landau levels. At these resonances, the effective cross section typically exceeds the classical Thomson value by over 2 orders of magnitude. Near and above the quantum critical magnetic field of 44.13 TeraGauss, relativistic corrections must be incorporated when computing this cross section. This paper presents formalism for the QED magnetic Compton differential cross section valid for both subcritical and supercritical fields, yet restricted to scattered photons that are below pair creation threshold. Calculations are developed for the particular case of photons initially propagating along the field, mathematically simple specializations that are germane to interactions involving relativistic electrons frequently found in neutron star magnetospheres. This exposition of relativistic, quantum, magnetic Compton cross sections treats electron spin dependence fully, since this is a critical feature for describing the finite decay lifetimes of the intermediate states. The formalism employs both the Johnson and Lippmann (JL) wave functions and the Sokolov and Ternov (ST) electron eigenfunctions of the magnetic Dirac equation. The ST states are formally correct for self-consistently treating spin-dependent effects that are so important in the resonances. Relatively compact analytic forms for the cross sections are presented that will prove useful for astrophysical modelers.