We study electron spin transport and the spin Hall effect in a quantum waveguide in the presence of Rashba spin-orbit interaction (SOI) and a homogeneous in-plane magnetic field, as well as the energy dispersion relation and spin polarization properties of the electron states in the corresponding infinite system. A general expression for the spin polarization of the outgoing flux, resulting from an arbitrarily polarized injection source, is derived for the quantum waveguide without the assumption of spin-degenerate leads. We further derive constraints on the spin-resolved transmission coefficients by analyzing the symmetries of the system. These constraints will impose restrictions on the spin-dependent conductances as well as the spin polarization of the outgoing flux. For an applied in-plane magnetic field parallel to the waveguide, conductance oscillations as a function of the SOI strength are diminished when the Zeeman energy exceeds the Rashba energy, due to spin alignment with the applied magnetic field. Furthermore, the calculated spin and charge probability distributions in the SOI waveguide region show spin Hall patterns and zitterbewegung patterns, respectively. For an applied transverse in-plane magnetic field, the SOI induced effective magnetic field is parallel or antiparallel with the applied field. The conductance of the system is then dominantly determined by spin-conserved transport processes, and spin-flipped transport processes can occur only through interchannel scattering and are therefore greatly suppressed. Furthermore, no zitterbewegung pattern in the charge density distribution can be found in the SOI region when the electrons are injected from a lead with spin polarization along the applied in-plane transverse field direction. However, the spin polarization probability distribution can still show a spin Hall pattern in the SOI region and the sign of the spin polarization in the spin Hall pattern can be reversed by tuning the applied magnetic field strength. For the corresponding infinite waveguide systems with uniform SOI and in-plane magnetic field, we show that the interplay between the applied in-plane magnetic field and the SOI induced effective magnetic field, together with subband hybridizations, will create a wave-vector-dependent spin polarization of the Bloch states with rich features and that these features cannot be captured by employing a simple one-dimensional model.