A study of the antiferromagnet CeRh$_2$Si$_2$ by torque, magnetostriction, and transport in pulsed magnetic fields up to 50 Tesla and by thermal expansion in static fields up to 13 Tesla is presented. The magnetic field-temperature phase diagram of CeRh$_2$Si$_2$, where the magnetic field is applied along the easy axis $\mathbf{c}$, is deduced from these measurements. The second-order phase transition temperature $T_{N}$ and the first-order phase transition temperature $T_{1,2}$ (=~36 K and 26 K at zero-field, respectively) decrease with increasing field. The field-induced antiferromagnetic-to-paramagnetic borderline $H_c$, which equals 26 T at 1.5 K, goes from first-order at low temperature to second-order at high temperature. The magnetic field-temperature phase diagram is found to be composed of (at least) three different antiferromagnetic phases. These are separated by the first-order lines $H_{1,2}$, corresponding to $T_{1,2}$ at H=0, and $H_{2,3}$, which equals 25.5 T at 1.5 K. A maximum of the $T^2$-coefficient $A$ of the resistivity is observed at the onset of the high-field polarized regime, which is interpreted as the signature of an enhanced effective mass at the field-induced quantum instability. The magnetic field dependence of the $A$ coefficient in CeRh$_2$Si$_2$ is compared with its pressure dependence, and also with the field dependence of $A$ in the prototypal heavy-fermion system CeRu$_2$Si$_2$.