The dichotomy of commissioning Indigenous health and wellbeing program evaluations: What the Funder wants vs what the Community needs.

Abstract

Health Promotion Journal of AustraliaVolume 32, Issue 2 p. 149-151 EDITORIALFree Access The dichotomy of commissioning Indigenous health and wellbeing program evaluations: What the Funder wants vs what the Community needs Summer May Finlay, Corresponding Author Summer May Finlay Summer.Finlay@canberra.edu.au Faculty of Health, University of Canberra, Bruce, Australia School of Health and Society, University of Wollongong, Wollongong, Australia Correspondence Summer May Finlay, Faculty of Health, University of Canberra. Email: Summer.Finlay@canberra.edu.auSearch for more papers by this authorMargaret Cargo, Margaret Cargo Faculty of Health, University of Canberra, Bruce, AustraliaSearch for more papers by this authorJames A. Smith, James A. Smith Wellbeing and Preventable Chronic Diseases Division, Menzies School of Health Research, Darwin, Australia College of Arts and Science, University of Saskatchewan, Saskatchewan, CanadaSearch for more papers by this authorJenni Judd, Jenni Judd Centre for Indigenous Health Equity Research, School of Health Medical and Applied Science, Central Queensland University, Bundaberg, AustraliaSearch for more papers by this authorAmohia Boulton, Amohia Boulton Faculty of Health and Environmental Sciences, AUT, Auckland, New Zealand Health Services Research Centre, Victoria University of Wellington, Wellington, New ZealandSearch for more papers by this authorDennis Foley, Dennis Foley Faculty of Business, Government & Law, University of Canberra, Bruce, AustraliaSearch for more papers by this authorYvette Roe, Yvette Roe College of Nursing and Midwifery, Charles Darwin University, Brisbane, AustraliaSearch for more papers by this authorBronwyn Fredericks, Bronwyn Fredericks UQ Poche Centre for Indigenous Health, University of Queensland, Brisbane, Australia Faculty of Health and Behavioural Sciences, University of Queensland, Brisbane, AustraliaSearch for more papers by this author Summer May Finlay, Corresponding Author Summer May Finlay Summer.Finlay@canberra.edu.au Faculty of Health, University of Canberra, Bruce, Australia School of Health and Society, University of Wollongong, Wollongong, Australia Correspondence Summer May Finlay, Faculty of Health, University of Canberra. Email: Summer.Finlay@canberra.edu.auSearch for more papers by this authorMargaret Cargo, Margaret Cargo Faculty of Health, University of Canberra, Bruce, AustraliaSearch for more papers by this authorJames A. Smith, James A. Smith Wellbeing and Preventable Chronic Diseases Division, Menzies School of Health Research, Darwin, Australia College of Arts and Science, University of Saskatchewan, Saskatchewan, CanadaSearch for more papers by this authorJenni Judd, Jenni Judd Centre for Indigenous Health Equity Research, School of Health Medical and Applied Science, Central Queensland University, Bundaberg, AustraliaSearch for more papers by this authorAmohia Boulton, Amohia Boulton Faculty of Health and Environmental Sciences, AUT, Auckland, New Zealand Health Services Research Centre, Victoria University of Wellington, Wellington, New ZealandSearch for more papers by this authorDennis Foley, Dennis Foley Faculty of Business, Government & Law, University of Canberra, Bruce, AustraliaSearch for more papers by this authorYvette Roe, Yvette Roe College of Nursing and Midwifery, Charles Darwin University, Brisbane, AustraliaSearch for more papers by this authorBronwyn Fredericks, Bronwyn Fredericks UQ Poche Centre for Indigenous Health, University of Queensland, Brisbane, Australia Faculty of Health and Behavioural Sciences, University of Queensland, Brisbane, AustraliaSearch for more papers by this author First published: 04 April 2021 https://doi.org/10.1002/hpja.486 Editor: James Smith AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat There are multiple systemic structural barriers that result in health and wellbeing inequities experienced by Aboriginal and Torres Strait Islander peoples (hereafter referred to as Indigenous). The recent Overcoming Indigenous Disadvantage 2020 report highlights some of these barriers, emphasising the importance of addressing the social determinants and cultural determinants of health by delivering evidence-based and culturally responsive health programs and initiatives.1 While is it estimated that $5.9 billion (AUS) is spent annually on Indigenous health programs and services in Australia, there is scant evidence to illustrate what works (or does not work), for who and why.2, 3 A mapping exercise of Indigenous-specific programs in Australia reported that of 1082 programs delivered, over 90% were not evaluated to assess whether programs achieved their goals and objectives.3 Also, it is reported that only a few evaluations used culturally appropriate methods for assessing program effectiveness.3 Recently, there have been a growing number of robust evaluations that are culturally responsive, from which we can learn. These have important implications that can guide the commissioning of future evaluations. 1 COMMISSIONING SETS THE STAGE FOR INDIGENOUS HEALTH AND WELLBEING EVALUATIONS The deficiency of evaluations in number and quality is problematic and viewed as stemming from the lack of effective engagement of commissioners and evaluators with Indigenous people.4-7 Although it is recognised that commissioning is a process, often involving a contractual arrangement, there is no accepted definition of commissioning. Despite the lack of an agreed definition, they share some similarities, with most definitions featuring the terms of reference for the evaluation and criteria for selecting the evaluator, and associated processes. The commissioning process is guided by the commissioning organisation's policy frameworks and principles that set the terms of reference on opportunities for Indigenous leadership and engagement in decision-making.8 The commissioning of an evaluation also sets the stage for the type of evaluation that will be conducted, the questions that will be asked and whether and how Indigenous people are engaged in the evaluation. 2 A CALL TO ‘NOTHING ABOUT US WITHOUT US!’ Over the past decade, there have been numerous government reports and Indigenous organisations calling for genuine partnership with commissioning agencies when assessing the impact of programs.4, 7-10 Indigenous communities have consistently stated, ‘nothing about us without us!’. A comprehensive review of health policies, including policies on commissioning found that the term 'Indigenous engagement' is often used and the importance of involving Indigenous community representatives in decision-making is frequently recognised. However, most policies do not go beyond 'consultation'. Mere consultation does not allow for Indigenous leadership in the commissioning process, therefore, tends to privilege non-Indigenous perspectives. This often leads to culturally unsafe evaluations and offers little in the way of benefit to the Indigenous communities who are the beneficiaries of the evaluated program. In instances where Indigenous leadership is recognised, it tends to be at the local level once the program and evaluation plans have already been identified by the commissioner.7 Such approaches to commissioning could be framed as 'participatory' or insofar as the commissioned evaluation approach makes provisions for the contracted evaluator for engaging local Indigenous people in the evaluation. 3 CHALLENGES TO EVALUATIONS FROM COMMISSIONING PROCESSES THAT DO NOT SUPPORT INDIGENOUS LEADERSHIP AND ENGAGEMENT Recent findings from a trans-Tasman study on culturally safe evaluation suggest that the way health programs and services are commissioned has a critical impact on whether communities are engaged in evaluation planning and implementation processes.11 Suppose funders were to listen to Indigenous people and ask the right evaluation questions at the initial program development stages? Then evaluations may be of more significant benefit to Indigenous communities. Unfortunately, current commissioning processes lack transparency and are often unclear how decisions made at the procurement or contracting stage influence what happens on the ground, in communities. Commissioned evaluations that do not support the active engagement of Indigenous people in leadership positions are prone to several problems. In the absence of authentic engagement, problems can arise from commissioners and evaluators who exert their decision-making authority in identifying the evaluation questions, preferences for the evaluation design and outcome measures.12-14 The lack of Indigenous peoples’ leadership and engagement in program design, delivery and evaluation is often driven by short timeframes in tendering processes.11, 15 It is not uncommon for providers to respond to a tender within 4-6 weeks and be required to design, deliver and evaluate programs to change well-entrenched attitudes and behaviours within 2 years. Many evaluations are commissioned after programs have been running for one or more years, which pose significant challenges in harmonising the evaluation's key performance indicators. Evaluations that do not build-in adequate Indigenous leadership and governance from the conception of the evaluation run significant risks associated with program duplication, poor coordination, harming the community, asking the wrong questions, accessing the wrong data sources and methodological unfeasibility.16 These issues also explain why, despite the level of funding and services over decades, there appears to be little improvement in health outcomes as evidenced by the Queensland Productivity Commission [Queensland Productivity Commission]. Consequently, many evaluations do not adequately capture Indigenous peoples’ lived experience with the program and subsequent measures of program success. International research reveals that commissioners of evaluation need more realistic timescales and openness to different methods and approaches, including more formative evaluation. They need to seek out help from experts and clients or professionals in the commissioning of evaluation.17 Indeed, globally it has been shown that commissioners of evaluations can often hold unrealistically high expectations for health improvements and cost savings as the outcome of integrated care initiatives.18 Evaluation questions and designs may be the problem as commissioners of evaluations jump to comparative or controlled research designs too quickly, ‘underestimating the value in process evaluations and embedded approaches that are able to describe the model, explore the variable penetration and fidelity, and undertake the necessary “sensemaking” before moving on to examining outcomes’.17, p3. 4 BUILDING ON EXISTING INDIGENOUS STRATEGIES, POLICIES AND FRAMEWORKS TO STRENGTHEN INDIGENOUS ENGAGEMENT IN COMMISSIONING Through the development of Indigenous-related strategies, policies and frameworks aimed at improving health outcomes, there has been some recognition that the commissioning of evaluations needs to improve. To overcome the scant evaluation evidence and poor financial return on the significant expenditure relating to Indigenous programs, the Productivity Commission, through a national consultation process, has developed a national Indigenous Evaluation Strategy (IES).19 The strategy highlights the need to strengthen both the evaluation capability and cultural capability of public servants engaged in commissioning evaluations of Indigenous programs and services. The IES identifies several 'leading practices' for agencies to centre Indigenous, perspectives and knowledge in the commissioning of evaluations. For planning, designing and conducting evaluations, agencies need to allocate sufficient time and resources for extensive engagement with Indigenous people during evaluations. Agencies also must strive to support Indigenous people's leadership in decisions related to data planning, collection and use. The Lowitja Institute has published an evaluation framework to improve Indigenous health.7 The publication recommends that tender processes, evaluation contracts and agreements align with the principles of working with Indigenous people and ethical frameworks. Although the National Health and Medical Research Council's Ethical conduct in research with Aboriginal and Torres Strait Islander peoples and communities: Guidelines for researchers and stakeholders has a focus on Indigenous engagement in the design and implementation of research, it can also be applied to the commissioning and conduct of evaluations.20 The Queensland Productivity Commission's Indigenous Inquiry Final Report also puts forward a range of models for engaging with communities and evaluation.21 If the principles and strategies outlined in the existing resources are implemented effectively, there is significant potential to strengthen the quality of evaluations in Indigenous health and wellbeing contexts. However, is this enough? Furthermore, are the implementation of these frameworks and strategies effectively monitored to ensure that Indigenous people are being engaged in leadership positions across all stages of a commissioned evaluation? We contend that a much greater focus needs to be paid to the commissioning of Indigenous health evaluations by commissioning bodies. We know very little about the commissioning models (eg, participatory, co-design), processes and approaches that support the evaluation of Indigenous programs. With no transparency, there can be little accountability. There needs to be a more nuanced understanding of the commissioning approaches employed to support Indigenous engagement and leadership, the most effective approaches and the contexts in which they would be most beneficial. Moreover, there is a need for deeper understandings of the complex interplay between commissioners, evaluators, service providers, other stakeholders and Indigenous communities. How such relationships are navigated, can greatly influence the conduct of the evaluation and ultimately the utility of evaluation findings, along with the actions taken or not taken over the longer term. It is now time to shift the Indigenous evaluation discourse to one which is focused on quality and comprehensiveness, not deficiency and mediocrity. Indeed, scholars have referred to this shortfall as the ‘gate keeping of research in Aboriginal communities’.22 Are you ready to join us in a new era of evaluating Indigenous health programs and services in a culturally-responsive way? REFERENCES 1 Steering Committee for the Review of Government Service Provision (SCRGSP). Overcoming Indigenous disadvantage: Key indicators 2020. Canberra, Australia: Productivity Commission. 2020: 4161. Google Scholar 2 Council of Australian Governments. National Indigenous Reform Agreement (Closing The Gap). Canberra, Australia: Council of Australian Governments; 2009: 110. Google Scholar 3Hudson S. Mapping the Indigenous program and funding maze. Sydney, Australia: The Centre for Independent Studies; 2016: 40. Google Scholar 4Thorpe A, Arabena K, Sullivan P, Silburn K, Rowley K. Engaging first peoples: A review of government engagement methods for developing health policy. Melbourne, Australia: The Lowitja Institute; 2016: 72. Google Scholar 5Priest N, Mackean T, Waters E, Davis E, Riggs E. Indigenous child health research: a critical analysis of Australian studies. Aust NZ J Public Health. 2009; 33(1): 55– 63. Wiley Online LibraryCASPubMedWeb of Science®Google Scholar 6Campbell D, Pyett P, McCarthy L. Community development interventions to improve Aboriginal health: Building an evidence base. Health Sociol Rev. 2007; 16(3-4): 304– 14. CrossrefWeb of Science®Google Scholar 7Kelaher M, Luke J, Ferdinand A, Chamravi D, Ewen S, Paradies Y. An evaluation framework to improve Aboriginal and Torres Strait Islander health. Melbourne, Australia: The Lowitja Institute; 2018: 84. Google Scholar 8 Department of Prime Minister and Cabinet. Indigenous advancement strategy evaluation framework. Canberra, Australia: Australian Government; 2018: 18. Google Scholar 9Dwyer J, O’Donnell K, Lavoie J, Marlina U, Sullivan P The Overburden Report: Contracting for Indigenous health services. Darwin, Australia: Cooperative Research Centre for Aboriginal Health; 2009 [cited 2021 Mar 19]. Available from: https://www.lowitja.org.au/content/Document/Lowitja-Publishing/Overburden-update-FINAL.pdfGoogle Scholar 10Dwyer J, Boulton A, Lavoie J, Tenbensel T, Cumming J. Indigenous peoples’ health care: new approaches to contracting and accountability at the public administration frontier. Public Manage Rev. 2013; 16(8): 1091– 112. CrossrefWeb of Science®Google Scholar 11Cargo M, Potaka-Osborne G, Cvitanovic L, Warner L, Clarke S, Judd J, et al. Strategies to support culturally safe health and wellbeing evaluation sin Indigenous settings in Australia and New Zealand: a concept mapping study. Int J Equity Health. 2019; 18(194): 1– 17. PubMedWeb of Science®Google Scholar 12Chi C, Schoon R, Mondaca A. Critical evaluation of international health programs: Reframing global health and evaluation. Int J Health Plann Mgmt. 2018; 33(2): 511– 23. Wiley Online LibraryPubMedWeb of Science®Google Scholar 13Sridhar D, Batniji R. Misfinancing global health: a case for transparency in disbursements and decision making. Lancet. 2008; 372(9644): 1185– 91. CrossrefPubMedWeb of Science®Google Scholar 14Evans S. A study of outcomes determination strategy in non-profit health and welfare programs evaluation. Eval J Australasia. 2017; 17(3): 23– 31. CrossrefGoogle Scholar 15Cargo M, Warner L. Evaluation of the Aboriginal Strengths-based Parental Engagement Program: Report prepared for the YWCA of Adelaide Inc. Adelaide, Australia: University of South Australia; 2013. Google Scholar 16Hudson S Evaluating Indigenous programs: A toolkit for change. Sydney, Australia: The Centre for Independent Studies; 2017: 58. Google Scholar 17Kumpunen S, Edwards N, Georghiou T, Hughes G Evaluating integrated care: Why are evaluations not producing the results we expect?. London, UK: Nuffield Trust; 2019 [cited 2021 March 17]. Available from: https://www.nuffieldtrust.org.uk/files/2019-11/the-challenges-of-evaluating-integrated-care-briefing-3.pdfGoogle Scholar 18Damery S, Flanagan S, Combes G. Does integrated care reduce hospital activity for patients with chronic diseases? An umbrella review of systematic reviews. BMJ Open. 2016; 6:e011952. CrossrefPubMedWeb of Science®Google Scholar 19 Productivity Commission Report. Indigenous evaluation strategy. Melbourne, Australia: Australian Government; 2020: 34. Google Scholar 20 National Health and Medical Research Council (NHMRC). Ethical conduct in research with Aboriginal and Torres Strait Islander peoples and communities: guidelines for researchers and stakeholders. Canberra, Australia: NHMRC; 2018 [cited 2021 Mar 19]. Available from: https://www.nhmrc.gov.au/about-us/resources/ethical-conduct-research-aboriginal-and-torres-strait-islander-peoples-and-communitiesGoogle Scholar 21 Queensland Productivity Commission. Final Report. Service delivery in remote and discrete Aboriginal and Torres Strait Islander communities. Brisbane, Australia: Queensland Productivity Commission; 2017 [cited 2021 Mar 19]. Available from https://qpc.blob.core.windows.net/wordpress/2018/06/Service-delivery-Final-Report.pdfGoogle Scholar 22Backhouse L. Gate keeping of research in Aboriginal Communities. Aborig Isl Health Work J. 1999; 23(5): 20– 5. Google Scholar Volume32, Issue2April 2021Pages 149-151 ReferencesRelatedInformation