Abstract Background Existing evaluation methods for assessing sexuality education programs may not be suitable for Comprehensive Sexuality Education (CSE) interventions, which are globally indicated as the most effective approach for promoting sexual health and preventing STIs. This holds particular significance in countries, such as Italy, where CSE is not integrated into school curricula. In a pilot project funded by the Italian Ministry of Health since 2019 (EduForIST), we aimed to develop a composite evaluation framework to test the CSE intervention we implemented in lower and upper secondary schools over the past 4 years. Methods Based on the latest available evidence, evaluating a CSE intervention requires addressing various dimensions. These include assessing the quality of programs and their implementation, as well as measuring outcomes and impact. Through iterative consultations with experts and providers, methods, criteria, and metrics were chosen and tested for each dimension. Results Program quality was evaluated by an expert advisory board. Instruments for implementation quality assessment included: satisfaction survey among students; reflective journals and SWOT analysis drawn up by educators delivering intervention, exploring their perceptions, practices, and attitudes; surveys among teachers and parents, measuring knowledge, attitude, and practice on CSE, and observed changes among adolescents. A four-arm, Solomon, cluster design study was designed to evaluate outcomes in terms of changes in students’ knowledge. Clinical and behavioral indicators in the short, medium and long term were selected as measures of the impact of the intervention. Conclusions In the field of CSE implementation research, a multi-component approach to evaluation is needed to provide a solid basis for informing policymakers. The involvement of providers, recipients and contextual adults is essential in evaluating CSE interventions. However, assessing long-term impact requires long time frame. Key messages • Assessing CSE interventions requires a multi-component evaluation approach, considering the impact on the personal and emotional development of individuals in order to sustain behavior changes. • The proposed evaluation framework may inform scientific community, professionals, and decision-makers on effectiveness of interventions in settings where CSE is not yet included in school curricula.