Non-significant p-values? Strategies to understand and better determine the importance of effects and interactions in logistic regression.

Zarina I Vakhitova,Clair L Alston-Knox,Yannick Griep

doi:10.1371/journal.pone.0205076

Zarina I Vakhitova, Clair L Alston-Knox + Show 1 more

Open Access

https://doi.org/10.1371/journal.pone.0205076

Copy DOI

Journal: PloS one	Publication Date: Nov 26, 2018
Citations: 28	License type: CC BY 4.0

Affiliation: Griffith University

Abstract

In the context of generalized linear models (GLMs), interactions are automatically induced on the natural scale of the data. The conventional approach to measuring effects in GLMs based on significance testing (e.g. the Wald test or using deviance to assess model fit) is not always appropriate. The objective of this paper is to demonstrate the limitations of these conventional approaches and to explore alternative strategies for determining the importance of effects. The paper compares four approaches to determining the importance of effects in the GLM using 1) the Wald statistic, 2) change in deviance (model fitting criteria), 3) Bayesian GLM using vaguely informative priors and 4) Bayesian Model Averaging analysis. The main points in this paper are illustrated using an example study, which examines the risk factors for cyber abuse victimization, and are further examined using a simulation study. Analysis of our example dataset shows that, in terms of a logistic GLM, the conventional methods using the Wald test and the change in deviance can produce results that are difficult to interpret; Bayesian analysis of GLM is a suitable alternative, which is enhanced with prior knowledge about the direction of the effects; and Bayesian Model Averaging (BMA) is especially suited for new areas of research, particularly in the absence of theory. We recommend that social scientists consider including BMA in their standard toolbox for analysis of GLMs.

Highlights

You can know the name of a bird in all the languages of the world, but when you’re finished, you’ll know absolutely nothing whatever about the bird
The aim of this paper is to delve beyond the automated analysis researchers often apply when implementing logistic regression, and to unpack the meaning held in inferences derived from four standard approaches to the statistical modelling of binary data, in regards to the interpretation of interactions
We demonstrate the limitations of the significance testing based on p-values and show how analyses in the Bayesian framework could help address these limitations

Summary

Introduction

You can know the name of a bird in all the languages of the world, but when you’re finished, you’ll know absolutely nothing whatever about the bird. I learned very early the difference between knowing the name of something and knowing something. There is no doubt that, due to the enhanced features it provides, data analysis using statistical modelling will, over time, replace statistical testing. This anticipated transition should—as applied disciplines shift to this more informative approach to data analysis—abate the tendency to report irrelevant p-values calculated using a rules-based framework. Models will be used to examine relationships between effects on a deeper level, allowing researchers to create or work within sound theoretical knowledge [3]

Objectives

Results

Conclusion