Excellence In Research For Australia Research Articles

A comprehensive examination of the relation of three citation-based journal metrics to expert judgment of journal quality

The academic and research policy communities have seen a long debate concerning the merits of peer review and quantitative citation-based metrics in evaluation of research. Some have called for replacing peer review with use of metrics for some evaluation purposes, while others have called for the use peer review informed by metrics. Whatever one's position, a key question is the extent to which peer review and quantitative metrics agree. In this paper we study the relation between the three journal metrics source normalized impact per paper (SNIP), raw impact per paper (RIP) and Journal Impact Factor (JIF) and human expert judgement. Using the journal rating system produced by the Excellence in Research for Australia (ERA) exercise, we examine the relationship over a set of more than 10,000 journals categorized into 27 subject areas. We analyze the relationship from the dimensions of correlation, distribution of the metrics over the rating tiers, and ROC analysis. Our results show that SNIP consistently has stronger agreement with the ERA rating, followed by RIP and then JIF along every dimension measured. The fact that SNIP has a stronger agreement than RIP demonstrates clearly that the increase in agreement is due to SNIP's database citation potential normalization factor. Our results suggest that SNIP may be a better choice than RIP or JIF in evaluation of journal quality in situations where agreement with expert judgment is an important consideration.

Still a burning issue: measuring screen production research

ABSTRACTValuing the production of screen works as research in the academy has become an urgent matter of research and scholarly equity facing the Australian screen production sector. Made up of filmmaker-academics this sector is collectively speaking out about a number of burning issues as evidenced in this article which deeply affect the kinds of scholarship and research enabled in the tertiary space. Representing the collective voices of a significant sample of Australian filmmaker-academics, this article offers some shared perspectives on the assessment of screen works by tertiary institutions created through creative practice research. Understanding what is occurring in the sector includes a deeper examination of three issues. The first issue arises from the Australian federal government metrics, used to classify research products such as films and screen works, called Excellence in Research for Australia and the Higher Education Research Data Collection. These two productivity metrics are disconnected when assessing films and screen works. The second issue is the development of criteria for valuing film and screen works as research, and the third issue relates to filmmaker-academic workload, as assessed by Australian institutions. When combined these issues present screen production academics with an overarching question – how are the production of screen works assessed and measured in and by tertiary institutions?

Information Sources and Indicators for the Assessment of Journal Reputation and Impact

ABSTRACTThis guide describes several information sources that can be used to assist faculty interested in quantitative and qualitative assessments of journal reputation and scholarly impact: Journal Citation Reports, Eigenfactor, Google Scholar Metrics, Elsevier Journal Metrics, Excellence in Research for Australia, Cabell’s International, Web of Science, Scopus, Google Scholar, and Beall’s List. It also introduces the indicators most often used to represent citation impact: impact factor, article influence score, eigenfactor, h5-index, source normalized impact per paper, impact per publication, and SCImago journal rank. Methods of assessing the influence of individual articles are also presented, along with strategies for the identification of predatory or low-quality journals.

Antipodean Reflections, Connections, and Perceptions

<scp>A</scp>ntipodean Reflections, Connections, and Perceptions

A Conceptual Matrix of Journalism as Research Two Decades after ‘Media Wars’

It is 20 years since John Hartley (1995) positioned journalism as the subject of academic research rather than as a research method in its own right. In 1999, Media International Australia devoted a themed edition to the debate over journalism in the academy (‘Media Wars’), which prompted further scholarly discourse over the role and location of journalism as a field of study. This article reassesses that debate in the light of the acknowledgement of journalism studies and journalism creative works in the Excellence in Research for Australia (ERA) system, the use of journalism methods as a research methodology and the development of conceptual paradigms for journalism as research. The article surveys the relationship between journalism and research over the ensuing two decades and proposes a conceptual matrix of the journalism–research nexus.

A network DEA quantity and quality-orientated production model: An application to Australian university research services

The motivation for this analysis is the recently developed Excellence in Research for Australia (ERA) program developed to assess the quality of research in Australia. The objective is to develop an appropriate empirical model that better represents the underlying production of higher education research. In general, past studies on university research performance have used standard DEA models with some quantifiable research outputs. However, these suffer from the twin maladies of an inappropriate production specification and a lack of consideration of the quality of output. By including the qualitative attributes of peer-reviewed journals, we develop a procedure that captures both quality and quantity, and apply it using a network DEA model. Our main finding is that standard DEA models tend to overstate the research efficiency of most Australian universities.

ERA 2012 Versus ERA 2010: Like the Curate's Egg … Good in Parts

ObjectiveThe Excellence in Research for Australia (ERA) assessment exercises undertaken in 2010 and 2012 were developed by the Australian Research Council (ARC) with the aim of identifying and promoting excellence in research across the full spectrum of research activity in Australia's higher education institutions.MethodThree fields of research are relevant to psychology and are denoted by the following codes: 1701 (psychology), 1702 (cognitive sciences), and 1799 (other psychology and cognitive sciences), as well as the combination of these, 17 (psychology and other cognitive sciences). In the first analysis of these data, all academic organisational units (AOUs) were compared across year (i.e., ERA 2010 and ERA 2012). The second analysis involved an examination of differences between AOUs in each of the 2010 and 2012 ERA data collections. Finally, the third analysis compared AOUs by the major groupings of universities that have emerged in the sector.ResultsThe comparison of the 2010 and 2012 data collections reveals an overall improvement in performance with most universities improving or at least holding their ground.ConclusionIt is concerning that a sizeable component of the production arising from the institutions reveals that only nearly half of the institutions meet a benchmark of at or above world standard. Some reasons for this performance are discussed, and suggestions are made for areas for improvement in the performance of the discipline going forward.

Research classification and the social sciences and humanities in Australia: (Mis)Matching organizational unit contribution and the impact of collaboration

The capacity to recognize the contribution of individual researchers and their organizational unit is likely to be at odds with the purpose of a national research assessment, due to the broader approach to identifying research strengths and its classification at disciplinary levels. The Australian research assessment exercise, Excellence in Research for Australia (ERA), is devised to determine research quality in this wider context. To explore the impact of research classification on the visibility of research contribution, this study analysed articles published by authors affiliated with two social sciences and humanities (SSH) fields—‘education’ and ‘language, communication and culture’. Article classification was examined for agreement with the authors’ affiliated organizational unit’s field. The influence of national and international co-authorship on article classification was also examined. Articles were distributed across different classification categories and a substantial proportion did not match with the authors’ organizational unit’s field. National and international co-authorship by education-affiliated authors was found to be significantly associated with publishing outside their organizational unit’s field. As the first investigation to focus on how research is distributed by the ERA’s classification scheme when applied to journal articles by SSH authors, the study provides empirical evidence of the challenges involved in recognizing the contribution of organizational units. This work builds on the existing literature relating to classification and research evaluation and has the potential to inform research managers of the complexities in setting strategic research priorities based on ERA outcomes.

Perspective: on the verge of a new ERA.

Despite its limitations, Excellence in Research for Australia was the right assessment tool at the right time, says Margaret Sheil.

FRONTLINE: Journalism as a research discipline

This article reviews recent debate about the performance and impact of the Excellence for Research in Australia (ERA) evaluations in 2010 and 2012 on the field of journalism research, in particular discussion of the relationship between research in journalism and that in the fields of communication, media and cultural studies. In response to that discussion and the initiation of the Frontline section in this journal (Bacon, 2012; Abplanalp, 2012; Gooch, 2012; Fitzgerald, 2013) it strongly argues that journalism is a distinct field of academic research practice. It identifies and briefly canvasses a range of methodological issues arising from Stuart Adam’s (1994) characterisation of journalism research as addressing the real, the present and the public (p. 13), and issues arising from shifting technologies and editorial/peer review processes. It indicates a range of methodological literature in cognate disciplines that can be used to ground journalism as a distinct research practice among the humanities and arts disciplines.

Open access to nursing journals: an audit of the 2010 ERA journal list

Objective: To determine the proportion of nursing journals that are fully open access or have some elements of openness, and hence are readily accessible by nurses in clinical settings. Design: A descriptive study, with interpretive analysis of existing data sources. Setting: Access to online journals from the perspective of a nurse as a consumer. Subjects: Nursing specific journals from the 2010 Excellence in Research for Australia (ERA) list of publications. Main outcome measure: The proportion of nursing journals with unrestricted online open access as of October 2011. Results: Of the 224 journals included in this research, 12% (27/224) had unrestricted open access to all published manuscripts, 39% (88/224) had partial or conditional access and 49% (109/224) had no elements of open access, and required the consumer to have access to a paid subscription. Conclusion: Approximately half (51%) of nursing journals surveyed have all or some articles that are open access. This is higher than other studies of open access research articles globally. However, barriers at an institutional level may need to be addressed to ensure nurses can be informed in their care of patients.

Education research Australia: a changing ecology of knowledge and practice

Processes of national research assessment, such as Excellence in Research for Australia (ERA) are a type of audit technology that confronts and steers established institutional identities and traditions. This nexus between policy and practice drives boundary work that diffracts prevailing policy logics, organisational practices, and habits of mind. We use this notion of ‘boundary work’ as an analytical lens for understanding the nature and effects of ERA in the Australian educational research space. This paper explains the methodology that informed the AARE–ACDE research reported in Strategic Capacity Building for Australian Educational Research. It documents the policy logic of ERA and the way it cuts across the established ecology of educational research, revealing social and symbolic work that is remaking the boundaries of educational research. We report on the historical trajectory of Australian educational research, the way ERA codes research outputs, and how educational researchers are repositioning in this shifting research space. We argue that there are specific loci of boundary work where capacity building in Australian educational research can make a difference to future educational knowledge building.

Education research in Australia: where is it conducted?

Research assessment exercises aim to identify research quantity and quality and provide insights into research capacity building strategies for the future. Yet with limited knowledge of the ecology of Australian educational research, there is little chance of understanding what research audits might contribute towards a capacity building agenda for such a complex field. This paper draws on secondary data analysis of research outputs submitted by 13 Australian higher education institutions to the Excellence in Research for Australia (ERA) 2010 and 2012 national research assessment exercises, to show where Australian educational research is conducted. Findings offer a profile of education researchers by location in academic organisational units within universities. By analyzing data not accessible through reported ERA data we were also able to present information about appointment profiles, specifically levels and type of appointment within universities, as well as data on institutional and geographic region, and patterns associated with type of outputs (books, book chapters, journal articles, conference papers and other outputs) and field of research. Analysis of the data reveals definitive shifts in the nature of the published outputs and in employment profiles of researchers and their location across university and regional groupings. Research audits are administrative processes that reshape institutional and disciplinary governance structures, policies, individual outputs, work practices and careers, but they are not the sum total of the field per se.

Reconfiguring research in New Zealand: education, politics and performativity

Introduced a decade ago, the Performance-Based Research Fund (PBRF) has had a substantial impact on research in New Zealand universities. The PBRF now provides the primary means for supporting research in tertiary education institutions, having replaced a system where funding was tied to student numbers. The PBRF is based on the principle of peer review, with expert panels in twelve different disciplinary groupings assigned the task of grading evidence portfolios (EPs) submitted by individual researchers. EPs comprise three main sections: (i) a list of research outputs (books, articles, conference papers, and so on), four of which are nominated by the researcher as his or her best; (ii) a section on ‘Peer Esteem’; and (iii) a component for ‘Contributions to the Research Environment’. The weightings in determining the final assessment are 70 % for the ‘outputs’ section, with 15 % each for the other two parts. In addition to the review of EPs via the quality evaluation exercise (accounting for 60 % of the total PBRF funding allocation), participating institutions and organisations are assessed on the basis of research degree completions and external research income. Three quality evaluation rounds—in 2003, 2006 (a partial round) and 2012—have been completed to date. Seddon and her colleagues have noted that, under Australia’s Excellence in Research for Australia (ERA) initiative, research in Education has been assessed as ‘below world standard’ (Seddon et al. 2012, p. 5). Education has also received a comparatively low subject area ranking in the latest round of the PBRF: 34th out of 42 (Tertiary Education Commission 2013, Appendix A, p. 89, Section 11). From the beginning, one of the key stated aims of the PBRF was to improve the quality of research in New Zealand. According to the Tertiary Education Commission, this goal is being achieved: ‘The measured research quality of the sector has increased over time: the new average quality score ... in 2012 is 4.66 compared to 4.40 in 2006 and 4.30 in 2003’ (p. 7). There was little change in the average quality score for Education

Living in a 2.2 world: from mapping to strategic capacity building for Australian educational research

The results of Australia’s first national research assessment, excellence for research in Australia, provoked considerable discussion about the state of educational research in Australia. Understanding the nature of this field of research became the focus for empirical research and analysis that was intended to inform strategic planning. This paper provides an overview of the articles in this special issue, bringing together research on Australian educational research and insights from international researchers who research the effects of research assessment in educational research in their own countries. These sources inform a commentary on Australian educational research and suggest an agenda for further action on strategic capacity building for Australian educational research.

Survival skills: the impact of change and the ERA on Australian researchers

This paper reports findings from a study that focused on the experiences of research-intensive academics in relation to the Excellence in Research for Australia (ERA) Framework. Interviews with academic staff at different career stages and across all academic faculties followed completion of a short survey in which respondents compared their publishing behaviour before and since implementation of the new Framework. Respondents were highly conscious that ERA had prompted a shift in publishing behaviour to meet often competing demands of individual research interests, institution, discipline and the international research community. Indeed, the study revealed academics to be positioned in contradictory ways in relation to their research and publishing, heightened by the instability of the Framework's assessment mechanisms. The experience of researchers up to, and including, the decision to abandon journal ranking provides valuable insights into the precarious and reactionary nature of academic research careers and the ability of both individuals and institutions to negotiate the rapid rate of change. These insights include tension between personal research priorities and ERA requirements, particularly in relation to writing for the most relevant audience, and concerns about the right to exercise academic freedom.

Editorial

The International Journal of Critical Indigenous Studies (IJCIS) now complements the recently launched National Indigenous Research and Knowledges Network (NIRAKN) in its efforts to build Indigenous research capacity. In this context the journal provides a platform for the research of Indigenous postgraduates, early- to mid-career researchers, and senior scholars. Indigenous scholars are therefore encouraged to submit their articles to future editions of the IJCIS, an ‘Excellence in Research for Australia’ (ERA) ranked journal

Environmental sciences research in northern Australia, 2000–2011: a bibliometric analysis within the context of a national research assessment exercise

This paper reports on a bibliometric analysis of environmental sciences research in northern Australia between 2000 and 2011. It draws on publications data for Charles Darwin University (CDU) and James Cook University (JCU) researchers to present a bibliometric profile of the journals in which they publish, the citations to their research outputs, and the key research topics discussed in the publications. Framing this analysis, the study explored the relationship between the two universities' publications and their `fit' with the environmental sciences field as defined by the Australian research assessment model, Excellence in Research for Australia (ERA). The Scopus database retrieved more records than Web of Science, although only minor differences were seen in the journals in which researchers published most frequently and the most highly cited articles. Strong growth in publications is evident in the 12 year period, but the journals in which the researchers publish most frequently differ from the journals in which the most highly cited articles are published. Many of the articles by CDU and JCU affiliated researchers are published in journals outside of the environmental sciences category as defined by Scopus and Web of Science categories and the ERA, however, the research conducted at each university aligns closely with that institution's research priorities.

An Uneven Playing Field: Rankings and Ratings for Economics in ERA 2010

In the evaluation of research quality conducted under Excellence in Research for Australia 2010, the sub-disciplines of econometrics and theory were rated more highly than the sub-disciplines of applied economics and other economics. The rating in each sub-discipline was benchmarked against a world standard, so the results suggest that Australian economists produce relatively better econometric or theory research than applied or other economics research. However, closer examination of the processes on which the ratings were based suggests built-in biases that favour theory and econometric research over applied and other economics research, leaving the relative quality of research in the various sub-disciplines open to question.

Landscape of ecological research in Australia: A bibliometric analysis of trends in research output and hotspots of research from 1991 to 2010

AbstractThe spatial distribution and density of scientists can have important implications for key aspects of scientific processes, such as innovation, networking, rates of knowledge exchange and success in large competitive grants. In this paper, we examine the research output of different research organizations and universities in Australia, with the aim of identifying hotspots of ecological research and how these hotspots have changed over the last 20 years. We used publications from 10 reputable peer‐reviewed international journals as a measure of research output. We identified a number of ecology hotspot clusters. Some clusters have developed significantly over the last 20 years, while others have declined in output over time. The University of Sydney, University of Queensland, James Cook University and Melbourne University had the largest output levels among universities. Results also showed large increases in output over the last 5‐year period (2006–2010), possibly because of the impending introduction of the Excellence in Research for Australia.