Research Assessment Measures Research Articles

Exploring the merits of research performance measures that comply with the San Francisco Declaration on Research Assessment and strategies to overcome barriers of adoption: qualitative interviews with administrators and researchers

BackgroundIn prior research, we identified and prioritized ten measures to assess research performance that comply with the San Francisco Declaration on Research Assessment, a principle adopted worldwide that discourages metrics-based assessment. Given the shift away from assessment based on Journal Impact Factor, we explored potential barriers to implementing and adopting the prioritized measures.MethodsWe identified administrators and researchers across six research institutes, conducted telephone interviews with consenting participants, and used qualitative description and inductive content analysis to derive themes.ResultsWe interviewed 18 participants: 6 administrators (research institute business managers and directors) and 12 researchers (7 on appointment committees) who varied by career stage (2 early, 5 mid, 5 late). Participants appreciated that the measures were similar to those currently in use, comprehensive, relevant across disciplines, and generated using a rigorous process. They also said the reporting template was easy to understand and use. In contrast, a few administrators thought the measures were not relevant across disciplines. A few participants said it would be time-consuming and difficult to prepare narratives when reporting the measures, and several thought that it would be difficult to objectively evaluate researchers from a different discipline without considerable effort to read their work. Strategies viewed as necessary to overcome barriers and support implementation of the measures included high-level endorsement of the measures, an official launch accompanied by a multi-pronged communication strategy, training for both researchers and evaluators, administrative support or automated reporting for researchers, guidance for evaluators, and sharing of approaches across research institutes.ConclusionsWhile participants identified many strengths of the measures, they also identified a few limitations and offered corresponding strategies to address the barriers that we will apply at our organization. Ongoing work is needed to develop a framework to help evaluators translate the measures into an overall assessment. Given little prior research that identified research assessment measures and strategies to support adoption of those measures, this research may be of interest to other organizations that assess the quality and impact of research.

Measuring accounting educators’ views on the teaching–research nexus (TRN): an international comparative study

ABSTRACT The relationship between teaching and research in the modern university has been the subject of vigorous scholarly enquiry in the education literature for several decades. Few international comparative studies are reported in the literature. This study compares and discusses the teaching–research nexus (TRN) in accounting in three international regions: the UK and Ireland (UKI); Australia and New Zealand (ANZ); and South Africa (SA). The purpose of this study is to provide evidence of that the TRN empirical model operates in the different regions, and more so to describe the TRN that remains challenging for the accounting academic profession globally. Positive synergies from the TRN are evident in all jurisdictions in this study but are largely ignored by academic institutions and professional accounting bodies (PABs) that influence the university accounting curriculum. Formal government research assessment measures, coupled by PABs pressures, result in accounting becoming more a teaching-led rather than research-informed academic discipline. This has implications for: the status of accounting academics; the public perception of the professional standing of accounting in society; and the learning experiences of accounting students.

Tools validated for assessment of mental health and cognition in older adults of Latin America and the Caribbean: A scoping review

AbstractBackgroundThe global population is ageing, with an estimated one in five people aged 60 by 2050. Latin America is one of the fastest ageing areas of the world and accurate clinical and research assessment measures of mental health in this ageing population are needed. It is not known which psychometric measures of older people’s mental health and cognition are validated in a Latin American context. This scoping review aims to address this question and outline current knowledge and research gaps.MethodBroad search terms were defined in English then translated and adapted to Spanish and Portuguese. Databases searched included Cinahl, Medline, Embase, Scielo, Scopus, Psychinfo, EBSCO and Web of Science. Inclusion criteria were validation or adaptation studies of tools for cognitive, mental health or psychosocial outcomes in older adults, conducted in Latin America or the Caribbean 1990‐2019. Titles and abstracts were reviewed blindly against search criteria using the RAYYAN application and conflicts resolved through discussion. PRISMA guidelines were utilized for reporting. Data extraction summarised psychometric procedures (reliability, construct/criterion validity, diagnostic accuracy, factor analysis etc), setting and participants.ResultsA total of 3805 articles were identified (3075 de‐duplicated) of which 526 underwent full‐text review. In total, 337 articles meeting the review criteria were identified and data were extracted. Psychometric evaluations of 138 different tools were identified, (90.3% conducted since 2006) though 46 were evaluated twice or less. Cognitive screening measures were the most numerous (n=107 45.1%) followed by depression and quality of life. Brazil had the largest number of studies (n=135). The most frequently evaluated tools were MMSE, MOCA, Geriatric Depression Scale (GDS) and Clock Drawing Test.ConclusionThe highest validation evidence was for cognitive measures. Most evidence relates to a small number of measures (cognition, depression and quality of life) originally developed in high‐income countries outside Latin America. The greatest evidence relates to, urban centres in Brazil.Though a large number of validation studies have been conducted, substantial research gaps remain. The majority of tools in current use have been insufficiently validated and their utility in clinical practice across different settings in Latin America remains unclear.

Comparison among journal impact factor, SCimago journal rank indicator, eigenfactor score and h5-index of environmental engineering journals

This paper reflects analyses the leading and reputable journal research quality indicators accepted by scientific and academic communities for evaluation and judgment of scientific journals in the field of environmental engineering. Selected research assessment measures are expected to highlight similarities within each indicator, indicate differences among alternative indicators, capture similar performance characteristics of highly cited journals, and show degree and type of correlations with indicators, and hence add significant journals informational values. Bibliometric indicators that are used to appraise quality ranking for journals use algorithms that are complex and other databases.The major objective of this research work is to assess presently applied (2015) quality indices of journal scientific impact with emphasis on Journal Impact Factor (JIF), Eigenfactor Score (ES), and SCImago Journal Rank indicator (SJR). Particular environmental engineering journals were chosen from their category within Web of Science. The 2015 JIFs and ESs are obtained from Journal Citation Report and the SJR from the SCImago Journal and country rank website.Chosen journals were listed and their associated information retrieved from their sites. Attempts were rendered to apply JIF, SJR and ES quality indexes for ranking and judging quality of environmental engineering journals. All journals were assessed in virtue of their JIF, ES and SJR. Likewise, correlations between indices were drawn through employment of Pearson’s and Spearman’s statistical correlations of the SPSS software. All journals examined have the premier standard of quality as being indexed in distinct and honored databases such as: Web of Science (WoS) and Scopus. JIF ranged between 8.328 and 0.129; ES varied between 0.05722 to as low as 0.00001, JSR oscillated between 2.322 and 0.127 and H5 swung between 121 to 5. A high Pearson’s (r) statistical correlation occurred between JIF and both H5 and SJR indicators for journals in this category (r = 0.859 and r = 0.853, respectively), while it is rather lower between JIF and ES values (r = 0.629). With respect to Spearsman’s rho statistical correlation a high correlation appeared between JIF and each of H5 and SJR indicators for journals in Environmental engineering (coefficient value of 0.867 and 0.862, respectively) and a nearly similar correlation between JIF and ES rankings (coefficient value = 0.851)

Bibliometric Rankings of Journals Based on the Thomson Reuters Citations Database

Virtually all rankings of journals are based on citations, including self citations by journals and individual academics. The gold standard for bibliometric rankings based on citations data is the widely-used Thomson Reuters Web of Science (2014) citations database, which publishes, among others, the celebrated Impact Factor. However, there are numerous bibliometric measures, also known as research assessment measures, based on the Thomson Reuters citations database, but they do not all seem to have been collected in a single source. The purpose of this paper is to present, define and compare the 16 most well-known Thomson Reuters bibliometric measures in a single source. It is important that the existing bibliometric measures be presented in any rankings papers as alternative bibliometric measures based on the Thomson Reuters citations database can and do produce different rankings, as has been documented in a number of papers in the bibliometrics literature.

Quality weighted citations versus total citations in the sciences and social sciences, with an application to finance and accounting

Purpose – Both journal self-citations and exchanged citations have the effect of increasing a journal’s impact factor, which may be deceptive. The purpose of this paper is to analyse academic journal quality and research impact using quality-weighted citations vs total citations, based on the widely used Thomson Reuters ISI Web of Science citations database (ISI). A new Index of Citations Quality (ICQ) is presented, based on quality-weighted citations. Design/methodology/approach – The new index is used to analyse the leading 500 journals in both the sciences and social sciences, as well as finance and accounting, using quantifiable Research Assessment Measures (RAMs) that are based on alternative transformations of citations. Findings – It is shown that ICQ is a useful additional measure to 2-year impact factor (2YIF) and other well-known RAMs for the purpose of evaluating the impact and quality, as well as ranking, of journals as it contains information that has very low correlations with the information contained in the well-known RAMs for both the sciences and social sciences, and finance and accounting. Practical implications – Journals can, and do, inflate the number of citations through self-citation practices, which may be coercive. Another method for distorting journal impact is through a set of journals agreeing to cite each other, that is, by exchanging citations. This may be less coercive than self-citations, but is nonetheless unprofessional and distortionary. Social implications – The premise underlying the use of citations data is that higher quality journals generally have a higher number of citations. The impact of citations can be distorted in a number of ways, both consciously and unconsciously. Originality/value – Regardless of whether self-citations arise through collusive practices, the increase in citations will affect both 2YIF and 5-year impact factor (5YIF), though not Eigenfactor and Article Influence. This leads to an ICQ, where a higher ICQ would generally be preferred to lower. Unlike 5YIF, which is increased by journal self-citations and exchanged citations, and Eigenfactor and Article Influence, both of which are affected by quality-weighted exchanged citations, ICQ will be less affected by exchanged citations. In the absence of any empirical evidence to the contrary, 5YIF and AI are assumed to be affected similarly by exchanged citations.

Robust Ranking of Journal Quality: An Application to Economics

The article focuses on the robustness of rankings of academic journal quality and research impact in general, and in economics, in particular, based on the widely-used Thomson Reuters ISI Web of Science citations database (ISI). The article analyzes 299 leading international journals in economics using quantifiable Research Assessment Measures (RAMs), and highlights the similarities and differences in various RAMs, which are based on alternative transformations of citations and influence. All existing RAMs to date have been static, so two new dynamic RAMs are developed to capture changes in impact factor over time and escalating journal self-citations. Alternative RAMs may be calculated annually or updated daily to determine When, Where, and How (frequently) published articles are cited (see Chang et al., 2011a–c). The RAMs are grouped in four distinct classes that include impact factor, mean citations, and non-citations, journal policy, number of high quality articles, journal influence, and article influence. These classes include the most widely used RAMs, namely, the classic 2-year impact factor including journal self-citations (2YIF), 2-year impact factor excluding journal self citations (2YIF*), 5-year impact factor including journal self citations (5YIF), Eigenfactor (or Journal Influence), Article Influence, h-index, and Papers Ignored–By Even The Authors (PI-BETA). As all existing RAMs to date have been static, two new dynamic RAMs are developed to capture changes in impact factor over time (5YD2 = 5YIF/2YIF) and Escalating Self-Citations (ESC). We highlight robust rankings based on the harmonic mean of the ranks of RAMs across the four classes. It is shown that emphasizing the 2YIF of a journal, which partly answers the question as to When published articles are cited, to the exclusion of other informative RAMs, which answer Where and How (frequently) published articles are cited, can lead to a distorted evaluation of journal quality, impact, and influence relative to the harmonic mean of the ranks.

JUST HOW GOOD ARE THE TOP THREE JOURNALS IN FINANCE? AN ASSESSMENT BASED ON QUANTITY AND QUALITY CITATIONS

The paper is concerned with ranking academic journal quality and research impact in Finance, based on the widely-used Thomson Reuters ISI (2013) Web of Science citations database (hereafter ISI). The paper analyses the 89 leading international journals in the ISI category of "Business–Finance" using quantifiable Research Assessment Measures (RAMs). The analysis highlights the similarities and differences in various RAMs, all of which are based on alternative transformations of journal citations and impact. Alternative RAMs may be calculated annually or updated daily to determine the citations frequency of published papers that are cited in journals listed in ISI. The RAMs include the classic 2-year impact factor including journal self citations (2YIF), 2-year impact factor excluding journal self citations (2YIF*), 5-year impact factor including journal self citations (5YIF), Immediacy including journal self citations, Eigenfactor (or Journal Influence), Article Influence (AI), h-index, Papers Ignored-By Even The Authors (PI-BETA), Self-citation Threshold Approval Rating (STAR), 5YD2 (namely, 5YIF divided by 2YIF), Escalating Self Citations (ESC) and Index of Citation Quality (ICQ). The paper calculates the harmonic mean (HM) of the ranks of up to 16 RAMs. It is shown that emphasizing 2YIF to the exclusion of other informative RAMs can lead to a misleading evaluation of journal quality and impact relative to the HM of the ranks. The analysis of the 89 ISI journals in Finance makes it clear that there are three leading journals in Finance, namely Journal of Finance, Journal of Financial Economics and Review of Financial Studies, which form an exclusive club in terms of the RAMs that measure journal quality and impact based on alternative measures of journal citations. The next two journals in Finance in terms of overall quality and impact are Journal of Accounting and Economics and Journal of Monetary Economics. As Accounting does not have a separate classification in ISI, the tables of rankings given in the paper are also used to rank the top 3 journals in the sub-category of Accounting in the ISI category of "Business – Finance", namely Journal of Accounting and Economics, Accounting Review, and Journal of Accounting Research.

Ranking Economics and Econometrics ISI Journals by Quality Weighted Citations

Abstract The paper analyses academic journal quality and impact using quality weighted citations that are based on the widely-used Thomson Reuters ISI Web of Science citations database (ISI). A recently developed Index of Citations Quality (ICQ), based on quality weighted citations, is used to analyse the top 276 Economics and top 10 Econometrics journals in the ISI Economics category using alternative quantifiable Research Assessment Measures (RAMs). It is shown that ICQ is a useful additional measure to the 2-Year Impact Factor (2YIF) and other well known RAMs available in ISI for the purpose of evaluating journal impact and quality, as well as ranking, of Economics and Econometrics journals as it contains information that has very low correlations with the information contained in alternative well-known RAMs. Among other findings, the top Econometrics journals have some of the highest ICQ scores in the ISI category of Economics.

How Should Journal Quality be Ranked?An Application to Agricultural, Energy, Environmental and Resource Economics

The Thomson Reuters ISI Web of Science citations database (hereafter ISI) category of Economics has one of the largest numbers of journals, at 304 (as of 2011) and 333 (as of 2013), of any ISI discipline, and hence has wide coverage. The paper analyses the leading international journals in the Economics sub-disciplines of Agricultural, Energy, Environmental and Resource Economics using quantifiable Research Assessment Measures (RAMs),and highlights the similarities and differences in alternative RAMs.The RAMs are based on alternative transformations of citations and influence taken from the ISI database. Alternative RAMs may be calculated annually or updated daily to answer the perennial questions as to When, Where and How (frequently) published papers are cited. The RAMs include the most widely used RAM, namely the classic 2-year impact factor including journal self citations (2YIF), 2-year impact factor excluding journal self citations (2YIF*), 5-year impact factor including journal self citations (5YIF), Immediacy (or zero-year impact factor (0YIF)), Eigenfactor, Article Influence, C3PO (Citation Performance Per Paper Online), h-index, PI-BETA (Papers Ignored - By Even The Authors), 2-year Self-citation Threshold Approval Ratings (2Y-STAR),HistoricalSelf-citation Threshold Approval Ratings (H-STAR), Impact Factor Inflation (IFI), and Cited Article Influence (CAI). As data are not available for 5YIF, Article Influence and CAI for one of the 20 journals considered, 13 RAMsare analysed for 19 highly-cited journals in Agricultural, Energy, Environmental and Resource Economics in the ISI category of Economics. The harmonic mean of the ranksof the 13 RAMs for the 19 highly-cited journals are also presented. It is shown that emphasizing the 2-year impact factor of a journal, which partly answers the question as to When published papers are cited, to the exclusion of other informative RAMs, which answer Where and How (frequently) published papers are cited, can lead to a distorted evaluation of journal impact and influence relative to the harmonic mean of the ranks. The 'age' effect of journals, that is, the number of years for which the journals have been included in ISI, on the RAMs is also examined to check whether the RAMs are being compared fairly.

Ranking Leading Econometrics Journals Using Citations Data from ISI and RePEc

The paper focuses on the robustness of rankings of academic journal quality and research impact of 10 leading econometrics journals taken from the Thomson Reuters ISI Web of Science (ISI) Category of Economics, using citations data from ISI and the highly accessible Research Papers in Economics (RePEc) database that is widely used in economics, finance and related disciplines. The journals are ranked using quantifiable static and dynamic Research Assessment Measures (RAMs), with 15 RAMs from ISI and five RAMs from RePEc. The similarities and differences in various RAMs, which are based on alternative weighted and unweighted transformations of citations, are highlighted to show which RAMs are able to provide informational value relative to others. The RAMs include the impact factor, mean citations and non-citations, journal policy, number of high quality papers, and journal influence and article influence. The paper highlights robust rankings based on the harmonic mean of the ranks of 20 RAMs, which in some cases are closely related. It is shown that emphasizing the most widely-used RAM, the two-year impact factor of a journal, can lead to a distorted evaluation of journal quality, impact and influence relative to the harmonic mean of the ranks. Some suggestions regarding the use of the most informative RAMs are also given.

WHAT DO EXPERTS KNOW ABOUT FORECASTING JOURNAL QUALITY? A COMPARISON WITH ISI RESEARCH IMPACT IN FINANCE

Experts possess knowledge and information that are not publicly available. The paper is concerned with forecasting academic journal quality and research impact using a survey of international experts from a national project on ranking academic finance journals in Taiwan. A comparison is made with publicly available bibliometric data, namely the Thomson Reuters ISI Web of Science citations database (hereafter ISI) for the Business–Finance (hereafter Finance) category. The paper analyses the leading international journals in Finance using expert scores and quantifiable Research Assessment Measures (RAMs), and highlights the similarities and differences in the expert scores and alternative RAMs, where the RAMs are based on alternative transformations of citations taken from the ISI database. Alternative RAMs may be calculated annually or updated daily to answer the perennial questions as to When, Where and How (frequently) published papers are cited (see Chang et al., 2011a,b,c). The RAMs include the most widely used RAM, namely the classic 2-year impact factor including journal self citations (2YIF), 2-year impact factor excluding journal self citations (2YIF*), 5-year impact factor including journal self citations (5YIF), Immediacy (or zero-year impact factor (0YIF)), Eigenfactor, Article Influence, C3PO (Citation Performance per Paper Online), h-index, PI-BETA (Papers Ignored — By even the Authors), 2-year Self-citation Threshold Approval Ratings (2Y-STAR), Historical Self-citation Threshold Approval Ratings (H-STAR), Impact Factor Inflation (IFI), and Cited Article Influence (CAI). As data are not available for 5YIF, Article Influence and CAI for 13 of the leading 34 journals considered, 10 RAMs are analysed for 21 highly-cited journals in Finance. The harmonic mean of the ranks of the 10 RAMs for the 34 highly-cited journals are also presented. It is shown that emphasizing the 2-year impact factor of a journal, which partly answers the question as to When published papers are cited, to the exclusion of other informative RAMs, which answer Where and How (frequently) published papers are cited, can lead to a distorted evaluation of journal impact and influence relative to the Harmonic Mean rankings. A linear regression model is used to forecast expert scores on the basis of RAMs that capture journal impact, journal policy, the number of high quality papers, and quantitative information about a journal. The robustness of the rankings is also analyzed.

Ranking journal quality by harmonic mean of ranks: an application to ISI statistics & probability

As the preponderance of journal rankings becomes increasingly more frequent and prominent in academic decision making, such rankings in broad discipline categories is taking on an increasingly important role. The paper focuses on the robustness of rankings of academic journal quality and research impact using on the widely-used Thomson Reuters ISI Web of Science citations database (ISI) for the Statistics & Probability category. The paper analyses 110 ISI international journals in Statistics & Probability using quantifiable Research Assessment Measures (RAMs), and highlights the similarities and differences in various RAMs, which are based on alternative transformations of citations and influence. Alternative RAMs may be calculated annually or updated daily to determine When, Where and How (frequently) published papers are cited (see Chang et al. (2011a, b, c), Chang et al. (2012)). The RAMs are grouped in four distinct classes that include impact factor, mean citations and non-citations, journal policy, number of high quality papers, and journal influence and article influence. These classes include the most widely used RAMs, namely the classic 2-year impact factor including journal self citations (2YIF), 2-year impact factor excluding journal self citations (2YIF*), 5-year impact factor including journal self citations (5YIF), Eigenfactor (or Journal Influence), Article Influence, h-index, PI-BETA (Papers Ignored - By Even The Authors), 5YD2 (= 5YIF/2YIF) as a measure of citations longevity, and Escalating Self Citations (ESC) as a measure of increasing journal self citations. The paper highlights robust rankings based on the harmonic mean of the ranks of RAMs across the 4 classes. It is shown that focusing solely on the 2-year impact factor (2YIF) of a journal, which partly answers the question as to When published papers are cited, to the exclusion of other informative RAMs, which answer Where and How (frequently) published papers are cited, can lead to a distorted evaluation of journal quality, impact and influence relative to the more robust harmonic mean of the ranks.

Citations and impact of ISI tourism and hospitality journals

The paper analyses the leading international journals in Tourism and Hospitality Research using quantifiable Research Assessment Measures (RAMs), highlights the similarities and differences in alternative RAMs, shows that several RAMs capture similar performance characteristics of highly cited journals, and shows that some other RAMs have low correlations with each other, and hence add significant informational value. Several RAMs are discussed for the Thomson Reuters ISI Web of Science database (hereafter ISI). Alternative RAMs may be calculated annually or updated daily to answer the questions as to When, Where and How (frequently) published papers are cited. The RAMs include the most widely used RAM, namely the classic 2-year impact factor including journal self citations (2YIF), 2-year impact factor excluding journal self citations (2YIF*), 5-year impact factor including journal self citations (5YIF), Immediacy (or zero-year impact factor (0YIF)), Eigenfactor, Article Influence, C3PO (Citation Performance Per Paper Online), h-index, PI-BETA (Papers Ignored - By Even The Authors), 2-year Self-citation Threshold Approval Ratings (2Y-STAR), Historical Self-citation Threshold Approval Ratings (H-STAR), Impact Factor Inflation (IFI), and Cited Article Influence (CAI). As data are not available for 5YIF, Article Influence and CAI for 11 of the 14 journals considered, 10 RAMs are analysed for 14 highly-cited journals in Tourism and Hospitality in the ISI category of Hospitality, Leisure, Sports and Tourism. Harmonic mean rankings of the 10 RAMs for the 14 highly-cited journals are also presented. A comparison of rankings is made for the h-index and C3PO for data from ISI and SCImago (2007) (see also SciVerse Scopus (2010)), which covers a broader range of journals, as well as a ranking of Tourism and Hospitality journals according to a subjective Aggregate Importance Rating. It is shown that emphasizing the 2-year impact factor of a journal, which partly answers the question as to When published papers are cited, to the exclusion of other informative RAMs, which answer Where and How (frequently) published papers are cited, can lead to a distorted evaluation of journal impact.

How are journal impact, prestige and article influence related? An application to neuroscience

The paper analyzes the leading journals in neurosciences using quantifiable research assessment measures (RAM), highlights the similarities and differences in alternative RAM, shows that several RAM capture similar performance characteristics of highly cited journals, and shows that some other RAM have low correlations with each other, and hence add significant informational value. Alternative RAM are discussed for the Thomson Reuters ISI Web of Science database (hereafter ISI). The RAM that are calculated annually or updated daily include the classic 2-year impact factor (2YIF), 5-year impact factor, immediacy (or zero-year impact factor), Eigenfactor score, article influence score, C3PO (citation performance per paper online), h-index, Zinfluence, PI-BETA (papers ignored by even the authors), 2-year and historical self-citation threshold approval ratings, impact factor inflation, and cited article influence (CAI). The RAM are analyzed for 26 highly cited journals in the ISI category of neurosciences. The paper finds that the Eigenfactor score and PI-BETA are not highly correlated with the other RAM scores, so that they convey additional information regarding journal rankings, that article influence is highly correlated with some existing RAM, so that it has little informative incremental value, and that CAI has additional informational value to that of article influence. Harmonic mean rankings of the 13 RAM criteria for the 26 highly cited journals are also presented. Emphasizing the 2YIF of a journal to the exclusion of other informative RAM criteria is shown to lead to a distorted evaluation of journal performance and influence, especially given the informative value of several other RAM.

COMPASS collaborative research Strand 1: assessment

Introduction and aimsStrand 1 develops assessment and outcome measures for use and intervention in research and clinical practice. We set out to improve the patient assessment platform, to improve comparable...

What makes a great journal great in the sciences? Which came first, the chicken or the egg?

The paper is concerned with analysing what makes a great journal great in the sciences, based on quantifiable Research Assessment Measures (RAM). Alternative RAM are discussed, with an emphasis on the Thomson Reuters ISI Web of Science database (hereafter ISI). Various ISI RAM that are calculated annually or updated daily are defined and analysed, including the classic 2-year impact factor (2YIF), 5-year impact factor (5YIF), Immediacy (or 0-year impact factor (0YIF)), Eigenfactor, Article Influence, C3PO (Citation Performance Per Paper Online), h-index, Zinfluence, PI-BETA (Papers Ignored--By Even The Authors), Impact Factor Inflation (IFI), and three new RAM, namely Historical Self-citation Threshold Approval Rating (H-STAR), 2 Year Self-citation Threshold Approval Rating (2Y-STAR), and Cited Article Influence (CAI). The RAM data are analysed for the 6 most highly cited journals in 20 highly-varied and well-known ISI categories in the sciences, where the journals are chosen on the basis of 2YIF. The application to these 20 ISI categories could be used as a template for other ISI categories in the sciences and social sciences, and as a benchmark for newer journals in a range of ISI disciplines. In addition to evaluating the 6 most highly cited journals in each of 20 ISI categories, the paper also highlights the similarities and differences in alternative RAM, finds that several RAM capture similar performance characteristics for the most highly cited scientific journals, determines that PI-BETA is not highly correlated with the other RAM, and hence conveys additional information regarding research performance. In order to provide a meta analysis summary of the RAM, which are predominantly ratios, harmonic mean rankings are presented of the 13 RAM for the 6 most highly cited journals in each of the 20 ISI categories. It is shown that emphasizing THE impact factor, specifically the 2-year impact factor, of a journal to the exclusion of other informative RAM can lead to a distorted evaluation of journal performance and influence on different disciplines, especially in view of inflated journal self citations.

WHAT MAKES A GREAT JOURNAL GREAT IN ECONOMICS? THE SINGER NOT THE SONG

Abstract This paper analyses what makes a great journal great in economics. Alternative research assessment measures (RAM) are discussed, with an emphasis on the Thomson Reuters Institute for Scientific Information (ISI) Web of Science database. ISI RAM that are calculated annually or updated daily are defined, including the classic 2-year impact factor (2YIF), 5-year impact factor (5YIF), immediacy (zero-year impact factor (0YIF)), eigenfactor score, article influence, citation performance per paper online, h-index, Zinfluence, PI-BETA (papers ignored - by even the authors) and two new RAM measures, self-citation threshold approval rating and impact factor inflation. The data are analysed for the most highly cited journals in economics, management, business and business–finance on the basis of 2YIF. In addition to evaluating research in the most highly cited journals in economics, management, business and business–finance, the paper evaluates alternative RAM, highlights similarities and differences in RAM criteria, finds that several RAM capture similar performance characteristics, and finds that immediacy and PI-BETA are not highly correlated with other RAM. Harmonic mean rankings of the 12 RAM criteria are also presented. Emphasizing 2YIF to the exclusion of other useful RAM criteria can lead to a distorted evaluation of journal performance and influence.

Great Expectatrics: Great Papers, Great Journals, Great Econometrics

The article discusses alternative Research Assessment Measures (RAM), with an emphasis on the Thomson Reuters ISI Web of Science database (hereafter ISI). Some analysis and comparisons are also made with data from the SciVerse Scopus database. The various RAM that are calculated annually or updated daily are defined and analyzed, including the classic 2-year impact factor (2YIF), 2YIF without journal self-citations (2YIF*), 5-year impact factor (5YIF), Immediacy (or zero-year impact factor (0YIF)), Impact Factor Inflation (IFI), Self-citation Threshold Approval Rating (STAR), Eigenfactor score, Article Influence, C3PO (Citation Performance Per Paper Online), h-index, Zinfluence, and PI-BETA (Papers Ignored – By Even The Authors). The RAM are analyzed for 10 leading econometrics journals and 4 leading statistics journals. The application to econometrics can be used as a template for other areas in economics, for other scientific disciplines, and as a benchmark for newer journals in a range of disciplines. In addition to evaluating high quality research in leading econometrics journals, the paper also compares econometrics and statistics, alternative RAM, highlights the similarities and differences of the alternative RAM, finds that several RAM capture similar performance characteristics for the leading econometrics and statistics journals, while the new PI-BETA criterion is not highly correlated with any of the other RAM, and hence conveys additional information regarding RAM, highlights major research areas in leading journals in econometrics, and discusses some likely future uses of RAM, and shows that the harmonic mean of 13 RAM provides more robust journal rankings than relying solely on 2YIF.

Great Expectatrics: Great Papers, Great Journals, Great Econometrics

The paper discusses alternative Research Assessment Measures (RAM), with an emphasis on the Thomson Reuters ISI Web of Science database (hereafter ISI). The various ISI RAM that are calculated annually or updated daily are defined and analysed, including the classic 2-year impact factor (2YIF), 5-year impact factor (5YIF), Immediacy (or zero-year impact factor (0YIF)), Eigenfactor score, Article Influence, C3PO (Citation Performance Per Paper Online), h-index, Zinfluence, and PI-BETA (Papers Ignored - By Even The Authors). The ISI RAM data are analysed for 8 leading econometrics journals and 4 leading statistics journals. The application to econometrics can be used as a template for other areas in economics, for other scientific disciplines, and as a benchmark for newer journals in a range of disciplines. In addition to evaluating high quality research in leading econometrics journals, the paper also compares econometrics and statistics, alternative RAM, highlights the similarities and differences in alternative RAM criteria, finds that several ISI RAM capture similar performance characteristics for the leading econometrics and statistics journals while the new PI-BETA criterion is not highly correlated with any of the other ISI RAM, and hence conveys additional information regarding ISI RAM, highlights major research areas in leading journals in econometrics, and discusses some likely future uses of RAM.