Policy recommendations from causal inference in physics education research

M B Weissman

doi:10.1103/physrevphyseducres.17.020118

Abstract

Sound educational policy recommendations require valid estimates of causal effects, but observational studies in physics education research sometimes have loosely specified causal hypotheses. The connections between the observational data and the explicit or implicit causal conclusions are sometimes misstated. The link between the causal conclusions reached and the policy recommendations made is also sometimes loose. Causal graphs are used to illustrate these issues in several papers from Physical Review Physics Education Research. For example, the core causal conclusion of one paper rests entirely on the choice of a causal direction although an unstated plausible alternative gives an exactly equal fit to the data.Received 24 January 2021Accepted 29 July 2021DOI:https://doi.org/10.1103/PhysRevPhysEducRes.17.020118Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.Published by the American Physical SocietyPhysics Subject Headings (PhySH)Research AreasEducational policyResearch methodologyPhysics Education Research

Highlights

A central goal of most physics education research (PER) is to find ways to teach better, i.e., to improve educational outcomes
It is inappropriate to represent a snapshot in time of a set of such traits by any acyclic directed mixed graphs (ADMGs) since effects generally have been flowing both ways between past values of the traits [2]. when I suggest alternatives to the causal interpretations used in these papers, I do not mean to imply that these alternatives are better and certainly not that they are right, since it is unlikely that any ADMG correctly represents the causal relations between snapshots of prolonged traits
The effects of out-class science and engineering activities (OCSE) on student attitudes were explored in a recent paper for which “...the primary goal of the current analysis is determining the impact of OCSE activities on physics identity”, i.e., estimating a causal effect [14]

Summary

INTRODUCTION

A central goal of most physics education research (PER) is to find ways to teach better, i.e., to improve educational outcomes. The purpose of this paper is to illustrate the need for better causal analysis in PER by using a few instances of papers published in this journal in which questionable procedures were used either to draw causal inferences from data or to draw policy recommendations from causal inferences or both. It is motivated by exchanges both with statisticians discontented with the level of causal reasoning in many social sciences and with colleagues in PER who were both uneasy about their lack of familiarity with. Ideally obtained by measuring effects of various interventions, will be needed in order to draw reliable causal conclusions to be used in making plans

BACKGROUND

CORRELATIONS DO NOT DETERMINE CAUSAL GRAPHS

Gender

Out-of-class activities

CAUSATION WITHIN AND OUTSIDE A GRAPH

DISCUSSION