The long-term survival rate of hemodialysis patients is limited by hemodialysis-related complications, which are attributed to the inappropriate level of relative blood volume (RBV) after treatment. For the conventional optical monitoring methods, the RBV is usually overestimated, especially when the RBV varies in a wide range. Here, we find that the scattering effect of blood on incident light is the dominant factor of nonlinear measurement errors according to the numerical simulation results. Then a computational model considering both the absorption and scattering effect of blood is proposed and applied to a relative blood volume monitoring system (RBVMS). Finally, the RBVMS is validated in a commercial hemodialysis machine, showing a relatively low measurement error of 6.9%. This result indicates the potential of the system for RBV monitoring during hemodialysis.