A new algorithm to evaluate the scallops left between consecutive tool tracks after five-axis machining of a complex-shaped part surface has been proposed. The algorithm has been developed for the ball-nose cutter. The novelty of the algorithm includes a variable plane to evaluate the effective tool profile and the part surface profile, the orientation of the tool as well as non-linear kinematics of the five-axis machine. The proposed algorithm has been specifically designed for and tested on the industrial Stereo lithography (STL) format representing complex shaped synthetic five-axis parts and a model of a crown of the molar tooth. The procedure has been tested against several modifications of the conventional curvature based method and the sphere intersection method. The ground truth is generated using the solid modeling engine of Vericut 8.2. The algorithm provides a tangible accuracy increase in terms of the average and the maximum error with regard to the reference methods.