Exploring Methodologies for Utilizing Click-Track Data Using Educational Data Mining and Evidence Centered Design in Online Professional Development Environments

Abstract

The opportunities for delivering effective PD has dramatically increased with the recent growth of web application capabilities. However, our methodologies for making meaningful inferences about these digital learning environments have remained limited. This dissertation argues that valid measurements of research constructs in online PD environments require three components to be successful: 1) a methodology for reliably recording user behavior, 2) an assessment framework for building the connection between constructs and observable behavior and 3) a statistical analysis approach that provides post-hoc methodologies for recognizing patterns in the observable behavior. The first manuscript in this dissertation conducts a review of the literature on methods used in a sample of existing studies and suggests that a combination of click-track data as the methodology for recording user behavior, Evidence Centered Design (ECD) as the assessment framework and Educational Data Mining (EDM) as the statistical analysis approach has the potential to provide insights in online PD environments. The second manuscript uses EDM methodologies to investigate construct assumptions and user behavior patterns through click-track data. The third manuscript uses a combination of ECD and EDM methodologies to build a measure for evaluating teacher engagement in an online PD environment. The dissertation provides case studies for the use of these combined methodologies, which show promise as a viable strategy for researching and understanding online PD environments. Insights and limitations of using click-track data and directions for further research are also discussed.