Introduction: As the field of natural orifice translumenal endoscopic surgery (NOTES) continues to progress, there is a need to objectively measure outcomes of training strategies and the impact of new devices on performance. The aim of this pilot study was to develop a global assessment tool specific for NOTES and to assess its reliability and construct validity. Methods: FITNESS (Formative Intraoperative Tool for NOTES Evaluation of Surgical Skills) was developed in consultation with expert NOTES practitioners and an educator. It consists of 6 items scored on a Likert scale from 1 to 5: access (A), navigation and orientation (NO), visualization and stabilization (VS), instrument manipulation and targeting (IMT), closure (C) and application of surgical principles (SP). Eight participants (4 novice and 4 experienced) were evaluated by two observers and in self assessment during performance of tasks in a live porcine model. Observers were all experienced NOTES surgeons and endoscopists. The procedure consisted of transgastric access, peritoneoscopy with identification of specified structures, liver biopsy and hemostasis, targeted peritoneal biopsy and closure of the gastrotomy using the Tissue Apposition System (TAS; Ethicon Endo-Surgery, Inc, Cincinnati, Ohio). The inter-rater reliabilities were calculated using intraclass correlation coefficients. Internal consistency was estimated using Cronbach's alpha. Observer scores for novice and experienced participants were compared using the Wilcoxon signed-rank test. Results: The inter-rater reliabilities for the total score between trained observers was 0.95 (95% CI 0.80-0.99), and between observers and self was 0.92 (0.76-0.98). The internal consistency of the FITNESS items was 0.96. The mean total score by the observers for novices was 17.3 ± 3.8, compared to 28.8 ± 1.3 for experienced operators (p=0.03). Conclusions: Preliminary data suggest that FITNESS is feasible, reliable and valid. These pilot data support its continued use for assessment of NOTES skills in the animate lab setting and as an educational tool in preparation for human procedures.