Substantive Researchers Research Articles

Improving training in methodology enriches the science of psychology.

Replies to the comment Ramifications of increased training in quantitative methodology by Herbet Zimiles on the current authors original article "Doctoral training in statistics, measurement, and methodology in psychology: Replication and extension of Aiken, West, Sechrest, and Reno's (1990) survey of PhD programs in North America". The current authors state that in their recent article, they reported the results of an extensive survey of quantitative training in all PhD programs in North America. They compared these results with those of a similar survey conducted 12 years earlier (Aiken, West, Sechrest, & Reno, 1990), and raised issues for the future methodological training of substantive and quantitative researchers in psychology. The authors then respond to Zimiles three questions.

Methods for the Behavioral, Educational, and Social Sciences: An R package

Methods for the Behavioral, Educational, and Social Sciences (MBESS; Kelley, 2007b) is an open source package for R (R Development Core Team, 2007b), an open source statistical programming language and environment. MBESS implements methods that are not widely available elsewhere, yet are especially helpful for the idiosyncratic techniques used within the behavioral, educational, and social sciences. The major categories of functions are those that relate to confidence interval formation for noncentral t, F, and chi2 parameters, confidence intervals for standardized effect sizes (which require noncentral distributions), and sample size planning issues from the power analytic and accuracy in parameter estimation perspectives. In addition, MBESS contains collections of other functions that should be helpful to substantive researchers and methodologists. MBESS is a long-term project that will continue to be updated and expanded so that important methods can continue to be made available to researchers in the behavioral, educational, and social sciences.

Inducing Score Reliability from Previous Reports: An Examination of Life Satisfaction Studies

The sample specific nature of reliability is often overlooked in substantive research, with investigators frequently referencing reliability estimates from previous studies as the basis for presuming the score integrity for their data. The purpose of the current study was to examine this practice, known as reliability induction, across studies using a popular life satisfaction (LS) measure, the Satisfaction with Life Scale. A comprehensive literature search was conducted, yielding a final sample of 196 studies using the instrument. Fifty-two studies involving 76 samples induced reliability coefficients from other sources. These studies were coded in terms of sample characteristics and variability estimates to derive comparisons in these two areas between the induced reports and the original studies. Results suggest that none of the studies inducing reliability had sufficient justification for induction. It was also found that estimating reliability coefficients for the data in hand has increased relative to induced reports and studies omitting reliability information. Practical implications for substantive researchers are discussed and suggestions for additional research are provided.

On Formal Theory and Statistical Methods: A Response to Carrubba, Yuen, and Zorn

The recent article by Carrubba, Yuen, and Zorn (2007) (CYZ) attempts to relate the strategic random utility models in Signorino (1999, 2002, 2003) and in Signorino and Yilmaz (2003) to existing game theory practice and to existing statistical techniques. It contributes to this literature by reminding us that comparative statics analysis can be applied to the equilibria of these models. There are a number of claims in CYZ, however, that require clarification. In particular, the article's primary claim is that comparative statics analysis, in combination with one of three proposed statistical estimators, provides a simpler alternative to methods previously advocated. This claim (or combination of claims) is incorrect. When one examines the procedure CYZ recommends, it is no simpler for substantive researchers than anything previously recommended. Moreover, none of the proposed estimators are new: they are exactly the same methods introduced in Signorino (1999, 2003), in Signorino and Yilmaz (2003), in Signorino, Walker, and Bas (2002), and in Bas, Signorino, and Walker (2007).

How to Use a Monte Carlo Study to Decide on Sample Size and Determine Power

A common question asked by researchers is, "What sample size do I need for my study?" Over the years, several rules of thumb have been proposed. In reality there is no rule of thumb that applies to all situations. The sample size needed for a study depends on many factors, including the size of the model, distribution of the variables, amount of missing data, reliability of the variables, and strength of the relations among the variables. The purpose of this article is to demonstrate how substantive researchers can use a Monte Carlo study to decide on sample size and determine power. Two models are used as examples, a confirmatory factor analysis (CFA) model and a growth model. The analyses are carried out using the Mplus program (Muthén& Muthén 1998).

Estimating La tent Variable Interactions and Quadratics: The State of This Art

Because there is little guidance for substantive researchers, the paper reviews the techniques for detecting latent variable interactions and quadratics. After examining plausible research situations where including these nonlinear variables might be appropriate, the paper describes the available detection techniques. Recent advances in nonlinear structural equation analysis are given particular attention, and suggestions for substantive researchers are made.

Differential Item Functioning: its Multidimensional Model and Resulting Sibtest Detection Procedure

A multidimensional model of differential item functioning (DIF) developed by Shealy and Stout is presented. It explains how individual items combine to produce differential test functioning (DTF) at the test level. Recent developments based on this approach, including development of the DIF/DTF detection procedure SIBTEST, are surveyed. The Shealy-Stout model not only offers insight into how DIF can occur, but suggests methods for investigating the root causes of DIF, which are useful to both substantive and statistical researchers of DIF. This new modeling paradigm offers the possibility of the proactive reduction of DIF in tests at the item manufacturing stage. SIBTEST is a nonparametric procedure that both tests for and estimates the amount of DIF in an item or set of items while controlling inflated Type I error by using a regression correction technique. It is shown to perform as well as and in many realistic situations better than other popular DIF assessment procedures. Recent modifications to the procedure demonstrate that it can be an effective tool for examining both crossing DIF and, through the use of kernel smoothing, local DIF. The procedure has been extended for use with tests containing polytomous items as well as tests that are intentionally multidimensional. Many real and simulated data analyses are presented.

Estimating latent variable interactions and quadratics: The state of this art

Because there is little guidance for substantive researchers, the paper reviews the techniques for detecting latent variable interactions and quadratics. After examining plausible research situations where including these nonlinear variables might be appropriate, the paper describes the available detection techniques. Recent advances in nonlinear structural equation analysis are given particular attention, and suggestions for substantive researchers are made.

Alternatives to classical statistical procedures: Introduction to the symposium.

AbstractMost classical statistical methods are founded on two assumptions, that the underlying distribution is normally distributed, and that the sample observations are independently and identically distributed. It did not take statisticians long to realize that these idealizations may not be true in all applied settings. In addition, the good performance of these classical procedures requires strict adherence to these assumptions. Consequently, alternatives to the classical procedures have been proposed by statisticians. The purpose of this symposium is to begin a dialogue between quantitative methodologists and substantive psychological researchers (who are voracious consumers of the classical procedures) about alternatives to the classical methods. The first three papers (Hunter & May, Zumbo & Zimmerman, and May & Hunter) will discuss a class of methods referred to as nonparametric statistics. Nonparametric procedures are an early and popular alternative to the classical procedures. Lind & Zumbo will discuss methods which are designed to deal with violations of normality. In closing, Thomas will discuss alternatives to classical statistical procedures when the second assumption is violated.The primary goal of the symposium is to expose generalists and specialists in the behavioural and social sciences to statistical procedures which are not usually part of their standard statistical repertoire. As well, we believe that it is important to begin communication among methodologists and substantive psychological researchers in a direct and clear manner (i.e., avoiding as much jargon as possible). Pursuant to these goals we solicited the participation of two statistically literate substantive psychological researchers who use clever data - analytic strategies in their substantive fields of cognition and development and have written an elementary introductory textbook on behavioural science statistics (Hunter & May). As well, some social science methodologists/statisticians (Lind, Thomas, Zimmerman, & Zumbo) were invited to present papers on recent developments in their work. In the spirit of the demolition of the Berlin Wall, we were interested in removing the disciplinary boundaries which usually isolate us.The dialogue is vital to both disciplines. Speaking for the methodologists, many of the problems we deal with are fueled by substantive issues and problems. As well, the substantive researchers are often the authors of social and behavioural science statistics textbooks. The authors of introductory elementary textbooks and instructors in introductory courses in Psychology departments almost inevitably fall short of being leading experts in statistics, but they are shapers of opinion and practice, having the effect of primacy in their favour. Yet, often advances in substantive areas are based upon the results of statistical tests. Therefore, it is important and vital for these substantive researchers to be as current as possible in their statistical knowledge.Topic of the SymposiumThe topic of the symposium was alternatives to classical statistical procedures, also referred to as alternatives to normal theory. We use the term classical statistics in the title because this is often used in the statistical literature and because of the historical precedence of these techniques. However, we prefer the term alternatives to normal theory not only because these techniques rely on the normal distribution but also because they are what are normally used (please excuse the pun).Normal theory includes common statistical tests using t, F, or z distributions such as ANOVA or regression. Also, multivariate techniques such as MANOVA, discriminant analysis, factor analysis, and covariance structural modelling/structural equation modelling are included. Most of these methods are the ordinary realizations of the Gaussian based (normal distribution) general linear model via least - squares estimation. …

Cross-Validation and Information Criteria in Causal Modeling

Many applications of causal modeling in marketing involve selection among several competing causal models. The author investigates whether common criteria for model selection such as cross-validation indices and information criteria are likely to lead to discovery of the correct population model. Guidance on the use of these selection criteria in practice is provided for substantive marketing researchers. Results indicate that the adequacy of cross-validation depends critically on the method used for sample-splitting. The author suggests the application of Snee's DUPLEX algorithm in this context. For situations in which the assumption of multinormally distributed variables is justified, information criteria are found to be highly appropriate for model selection, outperforming cross-validation methods in several respects.

Structural equation modeling in practice: A review and recommended two-step approach.

In this article, we provide guidance for substantive researchers on the use of structural equation modeling in practice for theory testing and development. We present a comprehensive, two-step modeling approach that employs a series of nested models and sequential chi-square difference tests. We discuss the comparative advantages of this approach over a one-step approach. Considerations in specification, assessment of fit, and respecification of measurement models using confirmatory factor analysis are reviewed. As background to the two-step approach, the distinction between exploratory and confirmatory analysis, the distinction between complementary approaches for theory testing versus predictive application, and some developments in estimation methods also are discussed.

Robustness of Estimators of the Squared Multiple Correlation and Squared Cross-Validity Coefficient to Violations of Multivariate Normality

A monte carlo experiment was conducted to evaluate the robustness of two estimators of the population squared multiple correlation ( R2p) and one estimator of the population squared cross-validity coefficient ( R2 cv) to a common violation of multivariate normality. Previous research has shown that these estimators are approximately unbiased when independent and dependent variables follow a joint multivariate normal distribution. The particular violation of multivariate normality studied here consisted of a dependent variable that may assume only a few discrete values. The discrete dependent variable was simulated by categorizing an underlying continuous variable that did satisfy the multivariate normality condition. Results illustrate the attenuating effects of categorization upon R2p and R2cv. In addition, the distributions of sample squared multiple correlations and sample squared cross-validity coefficients are affected by categorization mainly through the attenuations of R 2P and R2cv. Consequently, the formula estimators of R2p and R2cv were found to be as accurate and unbiased with discrete dependent variables as they were with continuous dependent variables. Substantive researchers who use categorical dependent variables, perhaps obtained by rating scale judgments, can justifiably employ any of the three estimators examined here.