In this paper, a method for the minimization of cutting forces and vibrations during precise ball end milling of hardened 55NiCrMoV6 steel is developed. The aim of this work concentrates on the optimal selection of surface inclination angle α and tool’s overhang l, which enables the minimization of cutting forces and vibrations in order to improve the machined surface quality. The experiment includes the measurement of cutting forces and acceleration of vibrations during the milling tests with variable input parameters. The next step focuses on the optimization of the ball end milling process with the consideration of the acquired signals. This procedure is carried out by the minimization of process responses with the application of signal to noise S/N ratio and grey relational analysis (GRA). Subsequently, the obtained optimal values of process input parameters are validated during the ball end milling tests involving the measurements of machined surface topographies. Research reveals that surface inclination angle and tool’s overhang have significant influence on generated forces and vibration values. Moreover, the selection of the optimal values of α and l enables significant improvement of machined surface quality.