BACKGROUND: This report describes the development and validation of a multiple choice examination on difficult airway management. It will be used as a formative assessment tool to monitor resident learning, detect deficits and provide feedback to residents about their educational progress. METHOD: We used an 8-step system to create the difficult airway exam. The steps are: (1) Identify target audience: clinical based anesthesia residents from 1st, 2nd and 3rd year (CA-1, CA-2, CA-3); (2) Develop specification table to identify topics in difficult airway management: knowledge of procedural skills, comprehension and application of airway techniques, American Society of Anesthesiologists’ difficult airway algorithm, medications for airway anesthesia, airway blocks, monitoring of airway management, oxygen delivery devices1,2; (3) Identify question writers; (4) Create a “how-to manual” on writing type A exam questions (selection of one best answer out of four); (5) Develop and use a checklist to ensure the exam follows the rules in the guidebook; (6) Identify external review board specialized in airway management to edit exam content; (7) Pilot the exam at four external sites–strict guidelines were followed when residents taking the exam (similar conditions to written boards);(8) Statistical analysis to validate test and assess discrimination between the groups. Three clinical anesthesiologists with international recognition in airway management took the exam and then checked their answers against the answer key and provided comments to improve the exam. RESULTS: The 112 question exam was piloted in March 2006 by 94 residents (n = 34 CA-1, 27 CA-2 and 33 CA-3) from University of Chicago, Children’s Hospital in South Carolina, University of Texas Medical School, and New York Mount Sinai Medical Center. The mean raw score at the four sites combined was 65 ± 10 for the group and 61± 6.6 for CA-1, 66 ± 7.5 for C2 and 68 ± 12.8 for CA-3 residents, p = .012 in analysis of variance. Multiple regression analysis controlling for residents’ age, sex and academic center indicated that CA-2 residents had a 6 point higher (p=.02) and CA 3 residents a 6.5 point higher score (p=.008) than CA-1 residents. The Kuder-Richardson reliability coefficient was 0.8.1 CONCLUSIONS: On the difficult airway exam, residents’ knowledge was demonstrated at 50% at four anesthesia departments (i.e. 65/112). The exam results indicate that the knowledge required in difficult airway management has yet to be achieved in our current method of clinical base teaching. Development of a validated test was achieved with good item performance following rigorous adherence to the 8 steps.