Measuring for management in science, technology, engineering, and mathematics learning ecosystems

Abstract

Rapid global expansion of multi-stakeholder ecosystems for learning in Science, Technology, Engineering, and Mathematics (STEM) demand close attention to the information infrastructure needed for sharing best practices and improving outcomes. The existing conceptual model of ecosystem development and elements needs to be translated into a metrological measurement model tailored to ecosystem management. An evaluation tool not designed from measurement principles included 38 questions applied across three years. An initial review of the instrument conducted before data were made available found multiple ambiguous items and rating categories requiring respondents to summarize diverse aspects of their experience in a single rating, with no capacity to reconstruct which aspects were included by any given respondent. Response data from 36 ecosystems in 2016, 38 in 2017, and 110 in 2018 produced more uncertainty than expected given the numbers of items, rating scale categories, and responses. No common factor structures across items could be identified. Stated expectations already on the record concerning STEM ecosystem development characteristics and focal partners, attributes, and goal areas provide a basis for an instrument redesign likely to result in meaningful measures advancing a theory of action fit for the purpose of STEM learning ecosystem management.