Developed as a plan-specific pre-treatment QA tool, Varian portal dosimetry promises a fast, high-resolution, and integrated QA solution. In this study, the agreement between predicted fluence and measured cumulative portal dose was determined for the first 140 patient plans at our Halcyon linear accelerator. Furthermore, the capability of portal dosimetry to detect incorrect plan delivery was compared to that of a common QA phantom. Finally, tolerance criteria for verification of VMAT plan delivery with Varian portal dosimetry were derived. All patient plans and the corresponding verification plans were generated within the Eclipse treatment planning system. Four representative plans of different treatment sites (prostate, prostate with lymphatic drainage, rectum, and head & neck) were intentionally altered to model incorrect plan delivery. Investigated errors included both systematic and random errors. Gamma analysis was conducted on both portal dose (criteria γ2%/2mm , γ2%/1mm , and γ1%/1mm ) and ArcCHECK measurements (criteria γ3%/3mm , γ3%/2mm , and γ2%/2mm ) with a 10% low-dose threshold. Performance assessment of various acceptance criteria for plan-specific treatment QA utilized receiver operating characteristic (ROC) analysis. Predicted and acquired portal dosimetry fluences demonstrated a high agreement evident by average gamma passing rates for the clinical patient plans of 99.90%, 96.64%, and 91.87% for γ2%/2mm , γ2%/1mm , and γ1%/1mm , respectively. The ROC analysis demonstrated a very high capability of detecting erroneous plan delivery for portal dosimetry (area under curve (AUC)>0.98) and in this regard outperforms QA with the ArcCHECK phantom (AUC ≈ 0.82). With the suggested optimum decision thresholds excellent sensitivity and specificity is simultaneously possible. Owing to the high achievable spatial resolution, portal dosimetry at the Halcyon can reliably be deployed as plan-specific pre-treatment QA tool to screen for errors. It is recommended to support the fluence integrated portal dosimetry QA by independent phantom-based measurements of a random sample survey of treatment plans.