Delivering adequate doses of radiotherapy (RT) for pancreatic cancer is challenging due to nearby organs at risk. Uncertainty about tumor location leads to large PTV expansions, which can increase toxicity. In this study, we used 4D MRI to assess motion of pancreatic tumor margins and their relationships with abdominal wall and diaphragmatic positions, with the goal of individualizing PTV margins and assessing the accuracy of abdominal wall or diaphragmatic surrogates for pancreatic tumor position. 17 patients with unresectable pancreatic cancer were enrolled in a prospective IRB-approved study and imaged free-breathing using non-contrast dynamic MRI on a 3T scanner. MR images were taken for one minute, at a rate of 3 images/sec, in single sagittal and coronal planes, using a 2D balanced fast field echo (BFFE) sequence. Images were acquired prior to and during treatment with chemoRT. Tumors were contoured with input from an abdominal MRI radiologist. Superior (sup), inferior (inf), anterior (ant), posterior (post), and lateral tumor margin motion were quantified using an in-house functional imaging analysis tools (FIAT) program. Average amplitudes and standard deviations of tumor margin motion were calculated. Pearson coefficients were used to assess correlation between motion of different tumor margins as well as diaphragmatic motion and abdominal excursion. Finally, a time-weighted distribution of tumor position relative to end-exhale was constructed. Pancreatic tumors moved inferiorly and anteriorly with inspiration as expected. In coronal slices, mean motion (std dev, range) of the sup and inf margins were 13 (4, 8–22) and 14 (6, 6–25) mm, respectively. In sagittal slices, mean motion (std dev, range) of the sup, inf, ant, and post margins were 14 (4, 6–25), 13 (4, 6–23), 6 (3, 2–13), and 4 (3, 0–9) mm. Lateral motion was negligible. Mean differences in motion between sequential scans for a single patient ranged from 2–6 mm. Movement of each tumor margin was poorly correlated with the others with the exception of the sup and inf margins on a single slice orientation (r = 0.62 and 0.72, p < 0.0001 for both), consistent with observed deformation of the pancreas during breathing. Despite similarity between the magnitude of diaphragmatic and sup or inf margin motion, diaphragmatic motion was poorly correlated with that of all tumor margins, with r values between −0.05 and 0.26 (p < 0.01). Abdominal wall excursion was also poorly correlated (r = −0.091 to −0.27, p < 0.0001). Distribution analysis revealed that the inf margin was >10 mm from end-exhale position 10–12% of the time, but it was >15 mm only 2–3% of the time. The sup margin spent 9–11% of the time >10 mm from end-exhale, but only 2% of the time further than 15 mm. Motion of the margins of pancreatic tumors is highly variable between patients and is generally larger than typically accounted for. The data suggest that there is substantial deformation with breathing, and motion of margins does not correlate well with abdominal wall or diaphragmatic motion. Treatment plans which rely on motion of a single fiducial or surrogates for tumor tracking should be used with caution. Instead, since tumors spend only a small proportion of time >10 mm from end exhale, treatment strategies employing time-weighted tumor position in dose calculation should be explored.