Background: The spatial distribution of charge and magnetization in the proton and neutron are encoded in the nucleon electromagnetic form factors. The form factors are all approximated by a simple dipole function, normalized to the charge or magnetic moment of the nucleon. The differences between the proton and neutron form factors and the deviation of GEn from zero are sensitive to the difference between up- and down-quark contributions to the form factors. Methods: We combine recent measurements of the neutron form factors with updated extractions of the proton form factors, accounting for two-photon exchange corrections and including an estimate of the uncertainties for all of the form factors to obtain a complete set of measurements up to Q^2 approximately 4 (GeV/c)^2. We use this to extract the up- and down-quark contributions which we compare to recent fits and calculations. Results: We find a large differences between the up- and down-quark contributions to G_E and G_M, implying significant flavor dependence in the charge and magnetization distributions. The rapid falloff of the ratio GEp/GMp does not appear in the individual quark form factors, but arises from a cancellation between the up- and down-quark contributions. We see indications that the down-quark contributions to the Dirac and Pauli form factors deviate from the suggested 1/Q^4 scaling behavior suggested by a previous analysis. Finally, while recent models provide a generally good qualitative description of the data, the down-quark contribution to G_E/G_M and F_2/F_1 are not reproduced by any of the models. Finally, we note that while the inclusion of recent $\gmn$ data from CLAS modifies the high-$Q^2$ slightly, the tension between these data and previous measurements at lower $Q^2$ has a more significant impact, suggesting the need for additional data in this region.