Abstract
Thirty experts in the assessment of the quality of Non-Traditional Research Outputs (NTROs) as academic research outputs were asked to rate the importance of 19 criteria that might be used in making these judgements. Analysis of responses identified four criteria where there is substantial agreement among the community of experts: (a) demonstrated familiarity in the research statement with the current state of knowledge in the relevant academic disciplines (very important); (b) demonstrated familiarity in the research statement with the current state of knowledge in the relevant industry (important); (c) evidence that the work has been engaged with by other academic researchers (relevant); (d) whether the NTRO creator is a substantive university staff member or an adjunct/honorary (unimportant). Fifteen other criteria either reached a less than ‘fair’ level of agreement, or larger numbers of respondents nominated ‘It depends’. Qualitative analysis of comments also revealed noteworthy disagreements in the expert community about how the criteria should be applied.
Highlights
Many countries have national systems established to evaluate the quality of academic research, each of which functions slightly differently (Rebora and Turri, 2013)
The analysis showed that of 19 possible criteria nominated by the Advisory Group to assess the quality of Non-Traditional Research Outputs’ (NTROs) as academic research outputs, there are four where there is general agreement among the community of experts as to their relative importance in assessing the quality of NTROs as academic research:
A critic ‘may not necessarily speak to the new knowledge that has come from the work’ says one expert, while another notes that ‘We will preference a review by a committee of academic peers’ rather than by a non-academic critic. In each of these contested criteria, we see that there exist in the community of NTRO assessors clear statements supporting their importance, and clear statements rejecting their importance, as well as a variety of positions along each continuum. The value of these data comes from showing us how the current community of expert assessors in Australia are engaging with debates about the nature of NTROs as academic research
Summary
Many countries have national systems established to evaluate the quality of academic research, each of which functions slightly differently (Rebora and Turri, 2013). This system includes a category of outputs named as ‘Non-Traditional Research Outputs’ (NTROs), which encompasses creative works, public exhibitions and events, or research reports produced for an external body, or portfolios of such works, that meets the Australian Research Council’s (2017) definition of academic research: the creation of new knowledge and/or the use of existing knowledge in a new and creative way to generate new concepts, methodologies, inventions and understandings. This [can] include the synthesis and analysis of previous research to the extent that it is new and creative. It might be hypothesised that the failure of these codes to follow the trend of other codes could be related to a lack of a clear, national, sector-wide understanding of the ways in which assessors are conceptualising the academic quality of NTROs
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
