Question Of Generalizability Research Articles

There is nothing WEIRD about basic research: The critical role of convenience samples in psychological science.

Attention to issues of sample diversity and generalizability has increased dramatically in the past 15 years, as psychological scientists have confronted the limitations of relatively homogeneous samples. Though this reckoning was perhaps overdue and has undoubtedly shined a light on some poor research practices, recommendations surrounding sample diversity are sometimes applied to research that does not aim for generalizability across peoples. In this article, I seek to promote discussion about when and why sample diversity and generalizability matter. In doing so, I address problems with language surrounding generalizability, the broader question of generalizability beyond samples, challenges for determining sufficient generalizability, and the inherent question of moderation in psychological science, given the reality of limited time and resources. I then discuss the important roles that basic research plays in understanding group differences, producing generalizable knowledge, and developing applied interventions. Finally, I address issues of equity surrounding sample diversity, emphasizing the distinction between WEIRD samples and convenience samples and the importance of convenience samples for globalizing psychological science. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

At What Cost? Is Technical Education Worth the Investment?

Abstract Career and technical education (CTE) has existed in the United States for over a century, and only in recent years have there been opportunities to assess the causal impact of participating in these programs while in high school. To date, no work has assessed whether the relative costs of these programs meet or exceed the benefits as described in recent evaluations. In this paper, we use available cost data to compare average costs per pupil in standalone high school CTE programs in Connecticut and Massachusetts to the most likely counterfactual schools. Under a variety of conservative assumptions about the monetary value of known educational and social benefits, we find that programs in Massachusetts offer clear positive returns on investment, whereas programs in Connecticut offer smaller, though mostly non-negative expected returns. We also consider the potential cost effectiveness of CTE programs offered in other contexts to address questions of generalizability.

Sequence similarity governs generalizability of de novo deep learning models for RNA secondary structure prediction.

Making no use of physical laws or co-evolutionary information, de novo deep learning (DL) models for RNA secondary structure prediction have achieved far superior performances than traditional algorithms. However, their statistical underpinning raises the crucial question of generalizability. We present a quantitative study of the performance and generalizability of a series of de novo DL models, with a minimal two-module architecture and no post-processing, under varied similarities between seen and unseen sequences. Our models demonstrate excellent expressive capacities and outperform existing methods on common benchmark datasets. However, model generalizability, i.e., the performance gap between the seen and unseen sets, degrades rapidly as the sequence similarity decreases. The same trends are observed from several recent DL and machine learning models. And an inverse correlation between performance and generalizability is revealed collectively across all learning-based models with wide-ranging architectures and sizes. We further quantitate how generalizability depends on sequence and structure identity scores via pairwise alignment, providing unique quantitative insights into the limitations of statistical learning. Generalizability thus poses a major hurdle for deploying de novo DL models in practice and various pathways for future advances are discussed.

Autism Spectrum Disorder and Attention-Deficit/Hyperactivity Disorder: Shared or Unique Neurocognitive Profiles?

Attention-deficit/hyperactivity (ADHD) and autism spectrum (ASD) disorders are commonly co-occurring conditions characterized by neurocognitive impairments. Few studies have directly compared neurocognitive profiles in ADHD and ASD and fewer still have controlled for comorbidity of ADHD and ASD. All direct comparisons have been in clinic samples, leaving the question of generalizability of results unaddressed. We compared neurocognitive performance in clinically ascertained ASD (n = 261) and ADHD (n = 423) cases and controls (n = 162), 6.0–17.9 years of age. We also compared ASD (n = 190) and ADHD (n = 926) cases ascertained in the community with controls (n = 14,842) of similar age. Using the stop-signal task (SST), we measured response inhibition (stop-signal reaction time–SSRT), sustained attention (defined as reaction time variability–RTV), and reaction time (RT). We controlled for comorbidity using ADHD and ASD trait scores and categorically-defined ADHD. Compared with controls, both clinic ADHD and ASD had significantly longer SSRT and RTV than controls and did not differ from each other. ADHD traits accounted for neurocognitive impairment in ASD, but not vice versa. There were no group differences for RT. Similar patterns of neurocognitive impairment were observed in the community sample. In the largest direct comparison of ADHD and ASD to date, we found impaired response inhibition and sustained attention in both disorders. However, neurocognitive impairment in ASD was almost completely accounted for by comorbid ADHD. Results generalized in the community sample indicating that referral bias alone did not drive results. Response inhibition and sustained attention likely play a role in ADHD and ASD.

Gender Ideology in Europe: Plotting Normative Types in a Multidimensional Space

Two recent articles (Grunow et al. in J Marriage Fam 80(1):42–60, 2018; Knight and Brinton in Am J Sociol 122(5):1485–1532) suggest that gender ideology is multidimensional. Such a finding is of utmost importance because, if robust, it can be used in future comparative (multilevel) research. However, these two articles present different results on which gender ideology profiles are dominant in Europe. Using the European Value Study, we replicate and extend these Latent Class analyses and address the question of generalizability and content- and criterion-related validity. We come to a five-cluster solution that not only synthesizes theoretically and empirically the results of the two articles, but also contributes to the literature by clarifying the place of these five gender ideology types in a multidimensional space. We suggest that in Europe five ‘worlds of norms’ exist that are mutually influenced by the general institutional context of welfare states.

Assessing fiscal distress in small county governments

Assessing fiscal distress in small county governments

Bi-Directional Gated Recurrent Unit Based Ensemble Model for the Early Detection of Sepsis.

Early prediction of sepsis is essential to give the patient timely treatment since each hour of delayed treatment has been associated with an increase in mortality. Current sepsis detection systems rely on empirical Clinical Decision Rules(CDR)s, which are based on vital signs that can be collected from the bedside. The main disadvantages of CDRs include questions of generalizability and performance variance when applied to the populations different from the groups used for derivation and often take years to develop and validate. This paper proposes a deep learning model using Bi-Directional Gated Recurrent Units(GRU), which uses a wide range of parameters that are associated with vitals, laboratory, and demographics of patients. The proposed model has an area under the receiver operating characteristic (AUROC) of 0.97, outperforming all the existing systems in the current literature. The model can handle the missing data, and irregular sampling intervals frequently present in medical records.Clinical relevance-The proposed model can be used to predict the onset of sepsis 6 hours ahead of time by the use of a machine learning algorithm. This proposed method outperforms the sepsis prediction machine learning models found in the current literature.

Reweighting Randomized Controlled Trial Evidence to Better Reflect Real Life - A Case Study of the Innovative Medicines Initiative.

Evidence from randomized controlled trials available for timely health technology assessments of new pharmacological treatments and regulatory decision making may not be generalizable to local patient populations, often resulting in decisions being made under uncertainty. In recent years, several reweighting approaches have been explored to address this important question of generalizability to a target population. We present a case study of the Innovative Medicines Initiative to illustrate the inverse propensity score reweighting methodology, which may allow us to estimate the expected treatment benefit if a clinical trial had been run in a broader real‐world target population. We learned that identifying treatment effect modifiers, understanding and managing differences between patient characteristic data sets, and balancing the closeness of trial and target patient populations with effective sample size are key to successfully using this methodology and potentially mitigating some of this uncertainty around local decision making.

Point-of-Care Bone Age Evaluation: The Increasing Role of US in Resource-limited Populations

Point-of-Care Bone Age Evaluation: The Increasing Role of US in Resource-limited Populations

Effects of notetaking instruction on intermediate and advanced L2 English learners: A quasi-experimental study

To prepare learners for the arduous tasks of simultaneously listening to and taking notes during (English medium instruction) EMI lectures, instructors have implemented explicit notetaking instruction with some degree of success. However, many such studies focus on improvements within the instructed group and do not compare results with those from control groups, leading to questions of generalizability. Thus, questions remain as to whether explicit notetaking instruction generates better results than uninstructed notetaking practice (e.g., when the teacher says “take notes").The present study involved control and intervention groups at the intermediate and advanced proficiency levels. The control groups received no explicit notetaking instruction, while the intervention groups received a 10-week period of scaffolded notetaking instructional cycle. Both groups completed lecture listening and notetaking activities prior to the explicit instruction. Student notes were collected and analyzed for “information units”, which are defined as the smallest item of information that on its own can be determined as true or false (Anderson, 2014). Statistical analysis of information unit scores show the notetaking instructional cycle had a significant impact on notetaking performance for the intervention groups. Pedagogic implications stemming from the findings are discussed, which have resonance for EMI and English for academic purposes (EAP) courses.

Climate for innovation and employee creativity

PurposeThis study proposes a moderated mediation model that examines the roles that openness to change and psychological capital (PsyCap) may play in the relationship between climate for innovation and employee creativity.Design/methodology/approachPath modeling analysis with software Mplus 7 is conducted to test our moderated mediation model.FindingsThe results show that climate for innovation promotes openness to change, which in turn encourages employee creativity, and PsyCap moderates not only the relationship between climate for innovation and openness to change but also the indirect effect of climate for innovation on employee creativity through openness to change.Research limitations/implicationsAlthough we attempt to avoid common method bias by collecting data in two waves, the six-month time interval separating the two waves of data collection may not be long enough to detect the causal relationship between climate for innovation and employee creativity. In addition, this study is conducted in companies located in China, which may raise the question of generalizability to other cultures.Originality/valueThe main contribution is building a moderated mediation model to uncover the potential mediating mechanism and boundary conditions associated with the influence of climate for innovation on employee creativity.

June 2018 Letter to the Editor-in-Chief.

Letter to the Editor-in-Chief of JOSPT as follows: "The Root Method as Fiction: The Original References" with Authors' Response J Orthop Sports Phys Ther 2018;48(6):514-515. doi:10.2519/jospt.2018.0202.

Examining personal values in extreme environment contexts: Revisiting the question of generalizability

Land-based extreme environments (e.g. polar expeditions, Antarctic research stations, confinement chambers) have often been used as analog settings for spaceflight. These settings share similarities with the conditions experienced during space missions, including confinement, isolation and limited possibilities for evacuation. To determine the utility of analog settings for understanding human spaceflight, researchers have examined the extent to which the individual characteristics (e.g., personality) of people operating in extreme environments can be generalized across contexts (Sandal, 2000) [1]. Building on previous work, and utilising new and pre-existing data, the present study examined the extent to which personal value motives could be generalized across extreme environments. Four populations were assessed; mountaineers (N =59), military personnel (N = 25), Antarctic over-winterers (N = 21) and Mars simulation participants (N = 12). All participants completed the Portrait Values Questionnaire (PVQ; Schwartz; 2) capturing information on 10 personal values. Rank scores suggest that all groups identified Self-direction, Stimulation, Universalism and Benevolence as important values and acknowledged Power and Tradition as being low priorities. Results from difference testing suggest the extreme environment groups were most comparable on Self-direction, Stimulation, Benevolence, Tradition and Security. There were significant between-group differences on five of the ten values. Overall, findings pinpointed specific values that may be important for functioning in challenging environments. However, the differences that emerged on certain values highlight the importance of considering the specific population when comparing results across extreme settings. We recommend that further research examine the impact of personal value motives on indicators of adjustment, group working, and performance. Information from such studies could then be used to aid selection and training processes for personnel operating in extreme settings, and in space.

Disciplinary Quandaries: A Metacommentary on the Relationship between Political Science and the Interdisciplinary Study of Asia

20 A & Q Kuhonta, Erik Martinez. 2011. The Institutional Imperative: The Politics of Equitable Development in Southeast Asia. Stanford, Calif.: Stanford University Press. Kuhonta, Erik Martinez, Dan Slater, and Tuong Vu, eds. 2008. Southeast Asia in Political Science: Theory, Region, and Qualitative Analysis. Stanford , Calif.: Stanford University Press. Scott, James C. 1985. Weapons of the Weak: Everyday Forms of Peasant Resistance. New Haven, Conn.: Yale University Press. Sidel, John T. 2006. Riots, Pogroms, Jihad: Religious Violence in Indonesia. Stanford, Calif.: Stanford University Press. Slater, Dan. 2010. OrderingPower:AuthoritarianLeviathansandContentious Politics in Southeast Asia. Cambridge: Cambridge University Press. Disciplinary Quandaries: A Metacommentary on the Relationship between Political Science and the Interdisciplinary Study of Asia Leela Fernandes The relationship between disciplinary practices and the interdisciplinary study of politics, culture, and society has long been a site of intellectual contestation within the U.S. academy. Such contestation has been particularly vigorous when it has been imbricated within relationships of power marked by inequalities like race, ethnicity, and gender or by historically specific global relationships of power between countries and regions of the world (Szanton 2004). Taken together, the commentaries in this feature provide an important analysis of the possibilities, limits, and challenges that the discipline of political science holds for the study of Asia and Asian diasporic communities and a productive avenue for an exploration of questions of disciplinarity, power, and knowledge. However, if the relationship between disciplinarity and interdisciplinarity is to be fully addressed—­both in relation to the study of Asia/Asian diasporas and in relation to the broader question of knowledge production—­it requires an exploration of the ways in which interdisciplinary fields of analysis (and their institutionalized sites) may also become disciplined by the cross-­ disciplinary formations that are dominant within them (Fernandes 2013; Miyoshi and Harootunian 2002). As the commentaries cogently argue, the turn away from “area studies” has had an impact on the discipline of political science. However, the question that then arises is how this A & Q 21 disciplinary trend has begun to impact the institutional presuppositions and conditions of knowledge production within the interdisciplinary study of Asia. A central constraint that has shaped political scientific scholarship on Asia and Asian diasporas is the institutionalization of a set of dominant norms regarding “methodological rigor.” The question of the effects of disciplinary methodological imperatives cuts across all of the commentaries on the state of the field. Whether they address the constraints of disciplinary norms that privilege quantitative scholarship (Lee; Wong), the devaluation of single-­ country “case study” research (Chung), or the formulaic dangers of mixed-­ methods approaches (Kuhonta; Lee), the commentaries point to the intellectual limits of a methodological rigidity and narrowness that produce chasms between political science and the study of Asia. The issue at hand is not that work that has drawn on quantitative or cross-­ national comparative approaches has not produced innovative and important scholarship on Asia or even that such approaches have not drawn on or contributed to in-­depth understandings of Asia. Mixed-­methods approaches in political science now often contain a fieldwork-­based component that may include interpretive methods that draw on oral interviews, ethnography, or discourse analysis. However, the methodological doctrine of political science presumes a hierarchy of knowledge practices that are tied to the kind of methods being deployed. Thus, for instance, in the mixed-­methods formula, interpretive methods are generally deployed to flesh out, extend, or deepen a research design that is defined by the foundational rigors of quantitative or cross-­national comparative methods. Such questions of methods are also fundamentally questions about intellectual legitimacy within the discipline. Methodological practices in effect become the disciplinary signposts of what kind of knowledge matters. Dominant conceptions of methodological rigor are inextricably bound up with epistemological judgments of what forms of knowledge are generalizable and therefore of significance to the discipline. However, the question of generalizability is itself encoded with long intellectual histories that have juxtaposed universal theories (historically coded as “Western” and “white”) with particular cases, contexts, and groups (historically coded as non-­Western or marked by race). Thus, whether we are speaking of the case of fields such as Asian American studies in political science or individual scholarly works that seek to analyze...

Is the clinical use of ice still relevant?

The use of cryotherapy, or the therapeutic application of cold, has become commonplace throughout much of the sports world. It is fairly typical to see a pitcher in Major League Baseball in the USA walking around with his entire arm wrapped in ice after he pitches, or to read testimonials of athletes in a variety sports about their experience in a cryochamber. Nevertheless, this propensity of use does not appear to be substantiated by research. Bleakley et al. (2004) concluded in their systematic review that insufficient evidence exists to support the use of cryotherapy clinically. Additionally, Hubbard & Denegar (2004) state that ‘the exact effect of cryotherapy on more frequently treated acute injuries has not been fully elucidated’. In this issue of Experimental Physiology, Lindsay et al. (2016) set out to elucidate the effects of repetitive cryotherapy in a mixed martial arts population. Using a population of 14 mixed martial artists, the scientists conducted an in vitro and an in vivo study to determine whether or not repetitive cryotherapy treatments affected the immune response to exercise. Their results were enlightening. In the in vitro and in vivo samples, T-cell and monocyte activation was decreased, whereas physical performance was not. The physical performance measurements included standing broad jump, press-ups and pull-ups. These data are meaningful because they get beyond the shortcomings of the studies discussed by Bleakley et al. (2004), more fully elucidating what is happening when someone treats with ice. A few questions arise from this study. First, are these results generalizable to an injured population? Many scientists have used delayed-onset muscle soreness as a model to study injury, yet it has not been proved that the inflammatory process associated with an injury truly acts the same way. The authors should be commended for using a population whose activities could more easily result in injury, but the question of generalizability remains. Second, do we want to decrease the immune response? The proponents against the use of cryotherapy use this as their main argument. If the inflammatory process is slowed, is healing slowed too? This is definitely something that must be studied further. Third, how closely do the treatment parameters reflect cryotherapy usage practices? In my experience, the use of cold water submersion tends to be something done regularly at the beginning of a sport season but gradually decreases over the first week or two. Admittedly, I have not spent any time in a mixed martial arts gymnasium, so I do not know whether the methods used in this article are customary in that setting. The authors’ results seem to indicate, however, that if regular cold water immersion is not used, should it be? When looking at the clinical use of cryotherapy as a whole, I believe confusion exists regarding when to use the modality. Furthermore, there is a lack of understanding that cryotherapy serves different purposes in certain situations. The article by Lindsay et al. (2016) is a step in the right direction to helping us understand what role cryotherapy plays in the treatment and rehabilitation of injury. Future studies may focus on the treatment of injuries experienced by this population as well as what temperature the water must be at to obtain the same benefits. Both studies would be enlightening, while helping us to understand this frequently used therapeutic modality.

Prediction of In-hospital Mortality in Emergency Department Patients With Sepsis: A Local Big Data-Driven, Machine Learning Approach.

Predictive analytics in emergency care has mostly been limited to the use of clinical decision rules (CDRs) in the form of simple heuristics and scoring systems. In the development of CDRs, limitations in analytic methods and concerns with usability have generally constrained models to a preselected small set of variables judged to be clinically relevant and to rules that are easily calculated. Furthermore, CDRs frequently suffer from questions of generalizability, take years to develop, and lack the ability to be updated as new information becomes available. Newer analytic and machine learning techniques capable of harnessing the large number of variables that are already available through electronic health records (EHRs) may better predict patient outcomes and facilitate automation and deployment within clinical decision support systems. In this proof-of-concept study, a local, big data-driven, machine learning approach is compared to existing CDRs and traditional analytic methods using the prediction of sepsis in-hospital mortality as the use case. This was a retrospective study of adult ED visits admitted to the hospital meeting criteria for sepsis from October 2013 to October 2014. Sepsis was defined as meeting criteria for systemic inflammatory response syndrome with an infectious admitting diagnosis in the ED. ED visits were randomly partitioned into an 80%/20% split for training and validation. A random forest model (machine learning approach) was constructed using over 500 clinical variables from data available within the EHRs of four hospitals to predict in-hospital mortality. The machine learning prediction model was then compared to a classification and regression tree (CART) model, logistic regression model, and previously developed prediction tools on the validation data set using area under the receiver operating characteristic curve (AUC) and chi-square statistics. There were 5,278 visits among 4,676 unique patients who met criteria for sepsis. Of the 4,222 patients in the training group, 210 (5.0%) died during hospitalization, and of the 1,056 patients in the validation group, 50 (4.7%) died during hospitalization. The AUCs with 95% confidence intervals (CIs) for the different models were as follows: random forest model, 0.86 (95% CI = 0.82 to 0.90); CART model, 0.69 (95% CI = 0.62 to 0.77); logistic regression model, 0.76 (95% CI = 0.69 to 0.82); CURB-65, 0.73 (95% CI = 0.67 to 0.80); MEDS, 0.71 (95% CI = 0.63 to 0.77); and mREMS, 0.72 (95% CI = 0.65 to 0.79). The random forest model AUC was statistically different from all other models (p ≤ 0.003 for all comparisons). In this proof-of-concept study, a local big data-driven, machine learning approach outperformed existing CDRs as well as traditional analytic techniques for predicting in-hospital mortality of ED patients with sepsis. Future research should prospectively evaluate the effectiveness of this approach and whether it translates into improved clinical outcomes for high-risk sepsis patients. The methods developed serve as an example of a new model for predictive analytics in emergency care that can be automated, applied to other clinical outcomes of interest, and deployed in EHRs to enable locally relevant clinical predictions.

Decentralizing for performance? A quantitative assessment of functional reforms in the German Länder

In the last 10 years, the governments of most of the German Länder initiated administrative reforms. All of these ventures included the municipalization of substantial sets of tasks. As elsewhere, governments argue that service delivery by communes is more cost-efficient, effective and responsive. Empirical evidence to back these claims is inconsistent at best: a considerable number of case studies cast doubt on unconditionally positive appraisals. Decentralization effects seem to vary depending on the performance dimension and task considered. However, questions of generalizability arise as these findings have not yet been backed by more ‘objective’ archival data. We provide empirical evidence on decentralization effects for two different policy fields based on two studies. Thereby, the article presents alternative avenues for research on decentralization effects and matches the theoretical expectations on decentralization effects with more robust results. The analysis confirms that overly positive assertions concerning decentralization effects are only partially warranted. As previous case studies suggested, effects have to be looked at in a much more differentiated way, including starting conditions and distinguishing between the various relevant performance dimensions and policy fields. Points for practitioners In multi-level systems, reformers of public administration have to decide which level is most appropriate for the implementation of a certain task. In the last years, reformers have often decided to decentralize state functions in the hope of creating an effective and efficient government. Yet, in reality, the claim to deliver public tasks better and, at the same time, cheaper in decentralized units proves unfeasible. We provide empirical evidence on the performance effects of decentralization and show that it has other advantages and other disadvantages compared to state service delivery. Reformers of public administration have to ponder these pros and cons in each particular case.

Consumer Nostalgia Literature Review and an Alternative Measurement Perspective

During the last two decades consumer nostalgia literature has experienced the growing amount of research, nonetheless, the nomological network in the area is still poorly established and fundamental questions of generalizability and measurement of nostalgia effects remain unanswered. This paper represents an attempt to comprehensively assess extant research in consumer nostalgia field, distinguish developments in the literature by summarizing the main findings of previous research and establishing theoretical trends. The analysis reveals that a number of demographic, social and psychological nostalgia antecedents, moderators and outcomes remain at the propositions level or lack the accumulated empirical quantitative support and validation from other studies. Therefore, specific recommendations regarding the development of nostalgia nomological network are provided to aid the continued theoretical and methodological improvements in the area. Since 1991 research in nostalgia has assumed that the correct measurement approach is a reflective one. This paper offers an alternative perspective for viewing and operationalizing nostalgia construct as a formative construct. Guidelines are summarized that aim to assist researchers with decision rules on whether to employ formative or reflective nostalgia measurement for future research. One of the main contributions of this study is to show the need for researchers to explicitly justify their choice of reflective or formative measurement models by supporting it with theoretical arguments and empirical evidence.

On the Generalizability of Experimental Results in Economics: With a Response to Commentors

Economists are increasingly turning to the experimental method as a means to estimate causal effects. By using randomization to identify key treatment effects, theories previously viewed as untestable are now scrutinized, efficacy of public policies are now more easily verified, and stakeholders can swiftly add empirical evidence to aid their decision-making. This study provides an overview of experimental methods in economics, with a special focus on developing an economic theory of generalizability. Given that field experiments are in their infancy, our secondary focus pertains to a discussion of the various parameters that they identify, and how they add to scientific knowledge. We conclude that until we conduct more field experiments that build a bridge between the lab and the naturally-occurring settings of interest we cannot begin to make strong conclusions empirically on the crucial question of generalizability from the lab to the field.

Observational Comparative Effectiveness Studies of Drug Therapies

Observational Comparative Effectiveness Studies of Drug Therapies