Multivariate Effect Sizes Research Articles

Vocational interests in the United States: Sex, age, ethnicity, and year effects.

Vocational interests predict educational and career choices, job performance, and career success (Rounds & Su, 2014). Although sex differences in vocational interests have long been observed (Thorndike, 1911), an appropriate overall measure has been lacking from the literature. Using a cross-sectional sample of United States residents aged 14 to 63 who completed the Strong Interest Inventory assessment between 2005 and 2014 (N = 1,283,110), I examined sex, age, ethnicity, and year effects on work related interest levels using both multivariate and univariate effect size estimates of individual dimensions (Holland's Realistic, Investigative, Artistic, Social, Enterprising, and Conventional). Men scored higher on Realistic (d = -1.14), Investigative (d = -.32), Enterprising (d = -.22), and Conventional (d = -.23), while women scored higher on Artistic (d = .19) and Social (d = .38), mostly replicating previous univariate findings. Multivariate, overall sex differences were very large (disattenuated Mahalanobis' D = 1.61; 27% overlap). Interest levels were slightly lower and overall sex differences larger in younger samples. Overall sex differences have narrowed slightly for 18-22 year-olds in more recent samples. Generally very small ethnicity effects included relatively higher Investigative and Enterprising scores for Asians, Indians, and Middle Easterners, lower Realistic scores for Blacks and Native Americans, higher Realistic, Artistic, and Social scores for Pacific Islanders, and lower Conventional scores for Whites. Using Prediger's (1982) model, women were more interested in people (d = 1.01) and ideas (d = .18), while men were more interested in things and data. These results, consistent with previous reviews showing large sex differences and small year effects, suggest that large sex differences in work related interests will continue to be observed for decades. (PsycINFO Database Record

The role of depressive symptomatology in peri- and post-menopause

ObjectivesThere is evidence that menopausal symptoms manifested at peri-menopause occur less frequently when compared to the symptoms experienced at post-menopause. The aim of this study was to investigate this and to test the hypothesis that depressive symptomatology mediates the relationship between menopausal stage and symptom frequency. MethodsThis cross-sectional study included 213 women (M age=52 years), of whom 125 were peri- and 88 post-menopausal. Measures comprised the Center for Epidemiologic Studies-Depression scale (CES-D) and the Women's Health Questionnaire (WHQ) vasomotor symptoms and somatic symptoms subscales. ResultsMultiple mediated regression analyses provided evidence that somatic symptoms and vasomotor symptoms were less frequent at post- compared to peri-menopause, and that these differences were mediated by depressive symptomatology. Multivariate effect sizes ranged from small to moderate, and univariate effect sizes were uniformly small with wide confidence intervals. ConclusionsThe frequency of vasomotor and somatic symptoms appears to increase with depressed affect. The management of symptoms could include interventions of a psychotherapeutic nature, which may offset this effect, particularly in women for whom depressive symptoms are a feature of the climacteric syndrome. The extent to which depression and the climacteric syndrome may be causally related to one another remains unclear and longitudinal research should further examine the mechanisms of this association.

Multivariate Common Language Effect Size.

To find a common language effect size of multivariate outcomes, we convert the standardized multivariate effect size (Mahalanobis distance) to a probability of a randomly selected subject from one population having a larger discriminant function than a randomly selected subject from another population. This probability is simple to calculate and comprehensible to laypeople. It can serve as the multivariate common language effect size to compare not only two groups but also more than two groups.

Multivariate Misgivings: Is D a Valid Measure of Group and Sex Differences?

In the study of group and sex differences in multivariate domains such as personality and aggression, univariate effect sizes may underestimate the extent to which groups differ from one another. When multivariate effect sizes such as Mahalanobis D are employed, sex differences are often found to be considerably larger than commonly assumed. In this paper, I review and discuss recent criticism concerning the validity of D as an effect size in psychological research. I conclude that the main arguments against D are incorrect, logically inconsistent, or easily answered on methodological grounds. When correctly employed and interpreted, D provides a valid, convenient measure of group and sex differences in multivariate domains.

Searchlight-based multi-voxel pattern analysis of fMRI by cross-validated MANOVA

Multi-voxel pattern analysis (MVPA) is a fruitful and increasingly popular complement to traditional univariate methods of analyzing neuroimaging data. We propose to replace the standard ‘decoding’ approach to searchlight-based MVPA, measuring the performance of a classifier by its accuracy, with a method based on the multivariate form of the general linear model. Following the well-established methodology of multivariate analysis of variance (MANOVA), we define a measure that directly characterizes the structure of multi-voxel data, the pattern distinctness D. Our measure is related to standard multivariate statistics, but we apply cross-validation to obtain an unbiased estimate of its population value, independent of the amount of data or its partitioning into ‘training’ and ‘test’ sets. The estimate D^ can therefore serve not only as a test statistic, but also as an interpretable measure of multivariate effect size. The pattern distinctness generalizes the Mahalanobis distance to an arbitrary number of classes, but also the case where there are no classes of trials because the design is described by parametric regressors. It is defined for arbitrary estimable contrasts, including main effects (pattern differences) and interactions (pattern changes). In this way, our approach makes the full analytical power of complex factorial designs known from univariate fMRI analyses available to MVPA studies. Moreover, we show how the results of a factorial analysis can be used to obtain a measure of pattern stability, the equivalent of ‘cross-decoding’.

Psychometric Properties of the Screen for Child Anxiety Related Emotional Disorders for Socially Anxious and Healthy Spanish Adolescents

Socially anxious and healthy Spanish adolescents were studied in order to test the psychometric properties of the Screen for Child Anxiety Related Emotional Disorders (SCARED). Confirmatory factor analyses were employed to test measurement invariance between these two populations, Cronbach's alphas were calculated to determine the reliabilities of the scales, and partial eta-square tests calculated the effect size of the differences between socially anxious and healthy adolescents and between the adolescent boys and girls. The psychometric properties of the SCARED were good, as demonstrated by having acceptable reliabilities (ranging from .75 - .41) and a moderate multivariate effect size (η(p)(2) = .08) between the adolescent boys and girls. Most importantly, it was demonstrated that the SCARED could differentiate between socially anxious and healthy Spanish adolescents as demonstrated by measurement invariance (χ(2) = 254.27, df = 1343, GFI = .884, AGFI = .872, RMR = .031) and the large effect size (η(p)(2) = .22) between the samples.

The distance between Mars and Venus: measuring global sex differences in personality.

BackgroundSex differences in personality are believed to be comparatively small. However, research in this area has suffered from significant methodological limitations. We advance a set of guidelines for overcoming those limitations: (a) measure personality with a higher resolution than that afforded by the Big Five; (b) estimate sex differences on latent factors; and (c) assess global sex differences with multivariate effect sizes. We then apply these guidelines to a large, representative adult sample, and obtain what is presently the best estimate of global sex differences in personality.Methodology/Principal FindingsPersonality measures were obtained from a large US sample (N = 10,261) with the 16PF Questionnaire. Multigroup latent variable modeling was used to estimate sex differences on individual personality dimensions, which were then aggregated to yield a multivariate effect size (Mahalanobis D). We found a global effect size D = 2.71, corresponding to an overlap of only 10% between the male and female distributions. Even excluding the factor showing the largest univariate ES, the global effect size was D = 1.71 (24% overlap). These are extremely large differences by psychological standards.SignificanceThe idea that there are only minor differences between the personality profiles of males and females should be rejected as based on inadequate methodology.

Clustering and meso-level variables in cross-sectional surveys: an example of food aid during the Bosnian crisis

BackgroundFocus groups, rapid assessment procedures, key informant interviews and institutional reviews of local health services provide valuable insights on health service resources and performance. A long-standing challenge of health planning is to combine this sort of qualitative evidence in a unified analysis with quantitative evidence from household surveys. A particular challenge in this regard is to take account of the neighbourhood or clustering effects, recognising that these can be informative or incidental.MethodsAn example of food aid and food sufficiency from the Bosnian emergency (1995-96) illustrates two Lamothe cluster-adjustments of the Mantel Haenszel (MH) procedure, one assuming a fixed odds ratio and the other allowing for informative clustering by not assuming a fixed odds ratio. We compared these with conventional generalised estimating equations and a generalised linear mixed (GLMM) model, using a Laplace adjustment.ResultsThe MH adjustment assuming incidental clustering generated a final model very similar to GEE. The adjustment that does not assume a fixed odds ratio produced a final multivariate model and effect sizes very similar to GLMM.DiscussionIn medium or large data sets with stratified last stage random sampling, the cluster adjusted MH is substantially more conservative than the naïve MH computation. In the example of food aid in the Bosnian crisis, the cluster adjusted MH that does not assume a fixed odds ratio produced similar results to the GLMM, which identified informative clustering.

Multivariate Effect Size Estimation: Confidence Interval Construction via Latent Variable Modeling

A latent variable modeling method is outlined for constructing a confidence interval (CI) of a popular multivariate effect size measure. The procedure uses the conventional multivariate analysis of variance (MANOVA) setup and is applicable with large samples. The approach provides a population range of plausible values for the proportion of generalized variance associated with group membership and permits at the same time testing of the underlying assumption of covariance matrix homogeneity. The method is generally applicable, is most readily used in empirical research with up to three simultaneously analyzed response variables, and is illustrated with an empirical example.

Fixed-Effects Meta-Analyses as Multiple-Group Structural Equation Models

Meta-analysis is the statistical analysis of a collection of analysis results from individual studies, conducted for the purpose of integrating the findings. Structural equation modeling (SEM), on the other hand, is a multivariate technique for testing hypothetical models with latent and observed variables. This article shows that fixed-effects meta-analyses with the following characteristics can be modeled in the SEM framework: (a) using any type of effect size; (b) including categorical and continuous moderators; and (c) including multivariate effect sizes. Empirical examples in LISREL syntax are used to demonstrate the equivalence between the meta-analytic and SEM approaches. Future directions for and extensions to this approach are discussed.

Meta‐Analysis in Criminal Justice and Criminology: What It is, When It's Useful, and What to Watch Out for

Meta‐analyses are becoming more common in the criminal justice and criminological literature. While much of the published work related to the technique is focused on the more technical aspects of the statistical methods that have been developed for meta‐analysis, less common are broader—and perhaps more critical—discussions concerning various issues associated with the method. Accordingly, this article presents an overview of meta‐analysis in the context of what have become three of the more important issues within the meta‐analysis literature in recent years: (1) the conditions under which meta‐analyses are, and are not, most useful, (2) the dilemma of whether or not to include unpublished work in the sample of studies to be analyzed, and (3) the choice of bivariate versus multivariate effect size estimates to be synthesized. The objective is to take these issues out of what has arguably been debates about technical orthodoxy and instead to place them into a broader research context within criminal justice and criminology.

The Relationship of Personality to Entrepreneurial Intentions and Performance: A Meta-Analytic Review

A set of meta-analyses were conducted to examine the relationship of personality to outcomes associated with two different stages of the entrepreneurial process: entrepreneurial intentions and entrepreneurial performance. A broad range of personality scales were categorized into a parsimonious set of constructs using the Five Factor model of personality. The results show that four of the Big Five personality dimensions were associated with both dependent variables, with agreeableness failing to be associated with either. Multivariate effect sizes were moderate for the full set of Big Five personality variables on entrepreneurial intentions (multiple R = .36) and entrepreneurial performance (multiple R = .31). Risk propensity, included as a separate dimension of personality, was positively associated with entrepreneurial intentions but was not related to entrepreneurial performance. These effects suggest that personality plays a role in the emergence and success of entrepreneurs.

On the Real Magnitude of Psychological Sex Differences

A comprehensive evolutionary theory of sex differences will benefit from an accurate assessment of their magnitude across different psychological domains. This article shows that mainstream research has severely underestimated the magnitude of psychological sex differences; the reason lies in the common practice of measuring multidimensional differences one dimension at a time, without integrating them into a proper multivariate effect size (ES). Employing the Mahalanobis distance D (the multivariate generalization of Cohen's d) results in more accurate, and predictably larger, estimates of overall sex differences in multidimensional constructs. Two real-world examples are presented: (1) In a published dataset on Big Five personality traits, sex differences on individual scales averaged d = .27, a typical ES conventionally regarded as “small.” However, the overall difference was D = .84 (disattenuated D = .98), implying considerable statistical separation between male and female distributions. (2) In a recent meta-analytic summary of sex differences in aggression, the individual ESs averaged d = .34. However, the overall difference was estimated at D = .75 – .80 (disattenuated D = .89–1.01). In many psychological domains, sex differences may be substantially larger than previously acknowledged.

Interval Estimates of Multivariate Effect Sizes

Monte Carlo methods were used to examine techniques for constructing confidence intervals around multivariate effect sizes. Using interval inversion and bootstrapping methods, confidence intervals were constructed around the standard estimate of Mahalanobis distance ( D2), two bias-adjusted estimates of D2, and Huberty’s I. Interval coverage and width were examined across conditions by adjusting sample size, number of variables, population effect size, population distribution shape, and the covariance structure. The accuracy and precision of the intervals varied considerably across methods and conditions; however, the interval inversion approach appears to be promising for D2, whereas the percentile bootstrap approach is recommended for the other effect size measures. The results imply that it is possible to obtain fairly accurate coverage estimates for multivariate effect sizes. However, interval width estimates tended to be large and uninformative, suggesting that future efforts might focus on investigating design factors that facilitate more precise estimates of multivariate effect sizes.

Effect-Size Measures and Meta-Analytic Thinking in Counseling Psychology Research

Effect sizes are critical to result interpretation and synthesis across studies. Although statistical significance testing has historically dominated the determination of result importance, modern views emphasize the role of effect sizes and confidence intervals. This article accessibly discusses how to calculate and interpret the effect sizes that counseling psychologists use most frequently. To provide context, the author presents a brief history of statistical significance tests. Second, the author discusses the difference between statistical, practical, and clinical significance. Third, the author reviews and graphically demonstrates two common types of effect sizes, commenting on multivariate and corrected effect sizes. Fourth, the author emphasizes meta-analytic thinking and the potential role of confidence intervals around effect sizes. Finally, the author gives a hypothetical example of how to report and potentially interpret some effect sizes.

Outcome evaluation of a community center-based program for mothers at high psychosocial risk

Objective This study reported the outcome evaluation of the “Apoyo Personal y Familiar” (APF) program for poorly-educated mothers from multi-problem families, showing inadequate behavior with their children. APF is a community-based multi-site program delivered through weekly group meetings in municipal resource centers. Method A total of 340 mothers referred by the municipal social services of Tenerife, Spain were assessed; 185 mothers participated in the APF program that lasted 8 months, and 155 mothers were in the control group. Pre-post test comparisons for the intervention group and post-test comparisons with the control group on self-rating measures of maternal beliefs, personal agency and child-rearing practices were performed. Multivariate tests, t tests and effect sizes (ES) were calculated to determine the program effectiveness on the outcome measures. Results Mothers’ support of nurturist and nativist beliefs and the reported use of Neglect-permissive and Coercive practices significantly decreased after program completion whereas the reported use of Inductive practices significantly increased. Increases in self-efficacy, internal control and role difficulty were also significant in relation to those of the control group. The program was especially effective for older mothers, with fewer children, living in a two-parent family, in urban areas and with either low or medium educational levels. Conclusion The program was very effective in changing the mothers’ perceived competences and modestly effective in changing their beliefs about child development and education and reported child-rearing practices. Changes in personal agency are very important for at-risk parents who feel helpless and with no control over their lives.

Multivariate Contrasts For Repeated Measures Designs Under Assumption Violations

Conventional and approximate degrees of freedom procedures for testing multivariate interaction contrasts in groups by trials repeated measures designs were compared under assumption violation conditions. Procedures were based on either least-squares or robust estimators. Power generally favored test procedures based on robust estimators for non-normal distributions, but was influenced by the degree of departure from non-normality, definition of power, and magnitude of the multivariate effect size.

MetaAnalysisCorr: An SAS/IML Program to Synthesize Correlation Matrices with Hunter and Schmidt, Hedges and Olkin, and Generalized Least Squares Approaches

Meta-analysis, the analysis of analyses, is a standard statistical procedure for summarizing research findings in behavioral sciences. The Hunter-Schmidt approach (Hunter & Schmidt, 1990) and the Hedges-Olkin (Hedges & Olkin, 1985) approach are the two most frequently used procedures. When dealing with multivariate cases, for instance, correlation matrices, these procedures are applied to individual correlation coefficients of the correlation matrix separately. A better approach, however, is to use the generalized least squares approach, which was proposed for multivariate effect sizes (e.g., Becker, 1992, 1995; Hedges & Olkin, 1985).

Testing Non-nested Multivariate Effect Size Models in Meta-Analysis

In modeling multiple effect sizes in meta-analysis, one often wants to evaluate two linear models for multivariate outcomes by including different sets of non-nested predictors for each outcome. This results in two non-nested multivariate effect size models. In this article, we show how to test the hypothesis that a non-nested model fits a set of predictors.

Influence of a complementary treatment with oral enzymes on patients with colorectal cancers--an epidemiological retrolective cohort study.

To evaluate the impact of postoperative treatment with an oral enzyme (OE) preparation given complementary to an antineoplastic therapy in patients with all stages of colorectal cancer. The design of this epidemiological study was a retrolective cohort analysis with parallel groups. Design and conduct of the study were performed to current standards for prospective, controlled clinical trials. Of a cohort of 1,242 patients with colorectal cancer (documented in 213 centres), 616 had received complementary treatment with OE (182 OE only, 405 other complementary drugs, 29 protocol violators) and 626 had not received OE (368 control only, 229 other complementary drugs, 29 protocol violators). Of 1,162 patients who had undergone primary surgery, 526 received adjuvant chemotherapy and 218 radiotherapy. The median follow-up time for the OE group was 9.2 months and for the control group 6.1 months. The primary test criterion of efficacy for OE treatment was the multivariate effect size of the changes from baseline of the disease- and therapy-associated signs and symptoms (nausea, vomiting, changes in appetite, stomach pain or stomach disorder, tiredness, depression, memory or concentration disorder, sleep disturbance, dizziness, irritability, dyspnoea at rest, dyspnoea during activity, headache, tumour pain, cachexia, skin disorders and infections). Tumour-related events, e.g. death, were evaluated by the number of events observed and time to event. Safety of treatment with OE was analysed in terms of number and severity of adverse events, their duration, treatment and outcome. A significant reduction in disease-associated signs and symptoms was observed in patients treated with OE alone, but not in those receiving OE in addition to other complementary treatments. Adverse reactions to chemo- and radiotherapy were diminished in all patients receiving OE. Analysis of survival did not demonstrate a reduced number of deaths in the OE group. However, a trend to prolongation of survival was demonstrated, particularly in the patients with disease stage Dukes' D, in the subgroup receiving OE in addition to other complementary treatments. Similar but less-pronounced trends were observed for disease stages Dukes' B and C. In the OE group, 21 of 616 patients (3.4%) experienced OE-associated adverse reactions, all of them mild to moderate gastrointestinal symptoms. Complementary treatment of colorectal cancer patients with OE improves their quality of life by reducing both the signs and symptoms of the disease and the adverse reactions associated with adjuvant antineoplastic therapies. This epidemiological retrolective cohort analysis provides evidence that patients may also benefit by a prolongation of survival time. OE were generally well tolerated.