Dislocations stored in heavily deformed materials play an important role in driving microstructure evolution. Here, we developed a full coupling model that concurrently couples the phase field method with crystal plasticity finite element analysis to study grain boundary (GB) migration under a plastic driving force. In our model, we describe multiple active grains in GB regions with crystal plasticity theory and use a weighted sum of their properties (i.e., stress and elastic/plastic potentials, etc.) to evaluate the plastic driving force for GB migration. The model can qualitatively capture the absorption of dislocations by mobile GBs through re-initialization of slip system resistances of newly active grains. A finite element based preconditioned Jacobian-free Newton-Krylov approach is used to simultaneously solve all the nonlinear partial differential equations for the coupled physics models. Determining model parameters and validation of the model are accomplished by simulating copper bicrystals and comparing the results to available experiments. This model provides a useful tool for effectively simulating GB migration in metals undergoing large plastic deformation. All the developments have been implemented in the MOOSE/MARMOT simulation package.