A complex machining model describing the coupled tool—workpiece dynamics subject to nonlinear regenerative cutting forces, instantaneous depth-of-cut and workpiece whirling due to material imbalance is presented. The workpiece is modeled as a system of three rotors, namely, unmachined, being machined and machined, connected by a flexible shaft, thus enabling the motion of the workpiece relative to the tool and tool motion relative to the machining surface to be three-dimensionally established as functions of spindle speed, depth-of-cut, rate of material removal and whirling. A rich set of nonlinear behaviors of both the tool and workpiece including period-doubling bifurcation and chaos signifying the extent of machining instability at various depth-of-cuts is observed. Results presented herein agree favorably with physical experiments reported in the literature. It is found that, at and up to certain ranges of depth-of-cuts, whirling is non-negligible if the fundamental characteristics of machining dynamics are to be fully understood. Additionally, contrary to one's intuition, whirling is found to have insignificant impact on tool motions.