BACKGROUND: Dedicated obstetric (OB) communication training is lacking, despite the wealth of uniquely sensitive and difficult clinical topics that are often unaddressed by conventional communication curricula. OBJECTIVE: To characterize the psychometric properties of an Obstetric Communication Assessment Tool (OCAT) for a novel OB communication module. METHODS: We developed four difficult OB Standardized Patient (SP) scenarios: Religious Beliefs (RB), Angry Father (AF), Maternal Smoking (MS), and Intimate Partner Violence (IPV). A trial run was performed with SPs and 21 third-year obstetrics and gynecology clerkship students. Five trained Standardized Patient Reviewers (SPRs) independently scored 24 randomized video-recorded encounters. Internal consistency (IC) was estimated with Cronbach's alpha. Inter-rater reliability (IRR) was estimated using the Intraclass Correlation Coefficient-2 (ICC-2) test. Systematic variability in reviewer responses was assessed using the Stuart-Maxwell test. RESULTS: IC was acceptable to excellent with alpha values (95% confidence intervals): RB 0.91 (0.86, 0.95), AF 0.76 (0.62, 0.87), MS 0.91 (0.86, 0.95), and IPV 0.94 (0.91, 0.97). IRR was unacceptable with ICC-2: RB 0.46 (0.40, 0.53), AF 0.48 (0.41, 0.54), MS 0.52 (0.45, 0.58), and IPV 0.67 (0.61, 0.72). Stuart-Maxwell analysis indicated systematic variability in reviewer scoring. CONCLUSION: We have developed the OCAT with high IC but unacceptable IRR. Our findings suggest that the OCAT effectively discriminates learner performance but systematic differences in SPR rigor require a revised training regimen to improve reliability. We propose the optimization of the OCAT as an essential step toward development of a focused OB communication module and discuss future directions of the project.