Health Sciences Admission Test Research Articles

Medical student selection process enhanced by improving selection algorithms and changing the focus of interviews in Australia: a descriptive study

The study investigates the efficacy of new features introduced to the selection process for medical school at the University of New South Wales, Australia: (1) considering the relative ranks rather than scores of the Undergraduate Medicine and Health Sciences Admission Test and Australian Tertiary Admission Rank; (2) structured interview focusing on interpersonal interaction and concerns should the applicants become students; and (3) embracing interviewers’ diverse perspectives. Data from 5 cohorts of students were analyzed, comparing outcomes of the second year in the medicine program of 4 cohorts of the old selection process and 1 of the new process. The main analysis comprised multiple linear regression models for predicting academic, clinical, and professional outcomes, by section tools and demographic variables. Selection interview marks from the new interview (512 applicants, 2 interviewers each) were analyzed for inter-rater reliability, which identified a high level of agreement (kappa=0.639). No such analysis was possible for the old interview since it required interviewers to reach a consensus. Multivariate linear regression models utilizing outcomes for 5 cohorts (N=905) revealed that the new selection process was much more effective in predicting academic and clinical achievement in the program (R2=9.4%–17.8% vs. R2=1.5%–8.4%). The results suggest that the medical student selection process can be significantly enhanced by employing a non-compensatory selection algorithm; and using a structured interview focusing on interpersonal interaction and concerns should the applicants become students; as well as embracing interviewers’ diverse perspectives.

The change from UMAT to UCAT for undergraduate medical school applicants: impact on selection outcomes.

To assess whether the change from the Undergraduate Medical and Health Sciences Admissions Test (UMAT; 1991-2019) to the University Clinical Aptitude Test (UCAT) for the 2020 New South Wales undergraduate medical degree intake was associated with changes in the impact of sex, socio-economic status and remoteness of residence, and professional coaching upon selection for interview. Cross-sectional study of applicants for the three NSW undergraduate medical programs for entry in 2019 (4114 applicants) or 2020 (4270); 703 people applied for both intakes. Applicants selected for interview were surveyed about whether they had received professional coaching for the selection test. Scores on the three sections of the UMAT (2019 entry cohort) and the five subtests of the UCAT (2020 entry); total UMAT and UCAT scores. Mean scores for UMAT 1 and 3 and for all four UCAT cognitive subtests were higher for men than women; the differences were statistically significant after adjusting for age, socio-economic status, and remoteness. The effect size for sex was 0.24 (95% CI, 0.18-0.30) for UMAT total score, 0.38 (95% CI, 0.32-0.44) for UCAT total score. For the 2020 intake, 2303 of 4270 applicants (53.9%) and 476 of 1074 interviewees (44.3%) were women. The effect size for socio-economic status was 0.47 (95% CI, 0.39-0.54) for UMAT, 0.43 (95% CI, 0.35-0.50) for UCAT total score; the effect size for remoteness was 0.54 (95% CI, 0.45-0.63) for UMAT, 0.48 (95% CI, 0.39-0.58) for UCAT total score. The impact of professional coaching on UCAT performance was not statistically significant among those accepted for interview. Women and people from areas outside major cities or of lower socio-economic status perform less well on the UCAT than other applicants. Reviewing the test and applicant quotas may be needed to achieve selection equity.

Trajectory of performance: The role of selection criteria on student achievement in a Bachelor of Oral Health programme.

To determine whether discontinuing the Undergraduate Medicine and Health Sciences Admissions Test (UMAT) in 2011 for selection and changes to the South Australian Certificate of Education (SACE) requirements in 2012 when the new Australian Tertiary Admissions Rank (ATAR) was introduced impacted on students' accumulative level of achievement and performance in the Bachelor of Oral Health (BOH) programme. Board of Examiner's course results for end of year performance were collected for BOH cohorts from 2006 to 2017 (n=347). Overall performance and achievement level attained were calculated and formed the dependent variables, Performance and Achievement. Group-based trajectory modelling (GBTM) characterised trajectories of yearly academic achievement and the likelihood of each case belonging to the latent group classified as individual group membership. Chi-square (or Fischer's exact) tests were conducted on groups and selected independent variables, using SAS 9.4. A 2-group academic Performance trajectory was selected as Consistent (n=290; 83.6%) and Inconsistent (n=57; 16.4%), and Achievement level (credit number) trajectory as Low (n=154; 44.4%) and High (n=193; 55.6%). Statistically significant findings in relation to achievement level were found; female students, those who enrolled after UMAT was discontinued, students enrolled since SACE changes, and those who nominated BOH as their first career preference, were more likely to achieve higher grades than their counterparts. Discontinuing the UMAT for selection and SACE changes have shown no adverse changes in students' level of achievement in the BOH programme, although enrolling in a course not their first preference was shown to affect performance level.

The economic costs of selecting medical students: An Australian case study.

The design of selection methods must balance, amongst a range of factors, the desire to select the best possible future doctors with the reality of our resource-constrained environment. Examining the cost of selection processes enables us to identify areas in which efficiencies may be gained. A cost description study was conducted based on selection for 2018 entry into medical school directly from secondary school. The perspectives of applicants, volunteer interviewers and the admitting institution were considered. Costs were modelled based on the Monash University (Australia) selection process, which uses a combination of secondary school matriculation score, aptitude test score (Undergraduate Medicine and Health Sciences Admission Test) and multiple mini-interview score. A variety of data sources were utilised, including bespoke surveys, audit data and existing literature. All costs are expressed in 2018 Australian dollars (AU$). Applicant behaviours in preparing for selection tests were also evaluated. A total of 381 of 383 applicants returned the survey. Over 70% of applicants had utilised commercial preparation materials. The median total cost to applicants was AU$2586 (interquartile range [IQR] AU$1574-3999), including costs to both prepare for and attend selection tests. Of 217 volunteer interviewers, 108 returned the survey. These were primarily health professional clinicians at a mid-career stage. The median total cost to interviewers was AU$452 (IQR AU$252-715) for participation in a half-day interview session, largely due to the loss of income. The cost to the admitting institution was AU$269 per applicant, accounted for by the costs of equipment and consumables (52%), personnel (34%) and facilities (14%). The costs of student selection for medical school are substantial. Understanding costs facilitates achievement of the objective of selecting the desired future medical workforce within the constraints of the resources available. Opportunities for change may arise from changes in applicant preparation behaviours, opportunities for economies of scale, and efficiencies driven by technological solutions.

Selecting top candidates for medical school selection interviews- a non-compensatory approach

BackgroundMedical schools apply a range of selection methods to ensure that admitted students succeed in the program. In Australia, selection tools typically include measures of academic achievement (e.g. the Australian Tertiary Admission Rank – ATAR) and aptitude tests (e.g. the Undergraduate Medicine and Health Sciences Admissions Test – UMAT). These are most commonly used to determine which applicants are invited for additional selection processes, such as interviews. However, no previous study has examined the efficacy of the first part of the selection process. In particular, are compensatory or non-compensatory approaches more effective in evaluating the outcomes of cognitive and aptitude tests, and do they affect the demographics of students selected for interview?MethodsThis study utilised data from consecutive cohorts of mainstream domestic students who applied to enter the UNSW Medicine program between 2013 to 2018. A compensatory ranked selection model was compared with a non-compensatory ranked model. Initially, ATAR marks and UMAT scores for each applicant were ranked within each cohort. In the compensatory model, the mean of the ATAR and UMAT ranks were used to determine the outcome. In the non-compensatory model, the lowest rank of ATAR and UMAT determined the outcome for each applicant. The impact of each model on the gender and socioeconomic status of applicants selected to interview was evaluated across all cohorts.ResultsThe non-compensatory ranked selection model resulted in substantially higher ATAR and UMAT thresholds for invitation to interview, with no significant effect on the socioeconomic status of the selected applicants.ConclusionsThese results are important, demonstrating that it is possible to raise the academic threshold for selection to medicine without having any negative impact on applicants from low socioeconomic backgrounds. Overall, the evidence gathered in this study suggests that a non-compensatory model is preferable for selecting applicants for medical student selection interview.

The efficacy of medical student selection tools in Australia and New Zealand.

To estimate the efficacy of selection tools employed by medical schools for predicting the binary outcomes of completing or not completing medical training and passing or failing a key examination; to investigate the potential usefulness of selection algorithms that do not allow low scores on one tool to be compensated by higher scores on other tools. Data from four consecutive cohorts of students (3378 students, enrolled 2007-2010) in five undergraduate medical schools in Australia and New Zealand were analysed. Predictor variables were student scores on selection tools: prior academic achievement, Undergraduate Medicine and Health Sciences Admission Test (UMAT), and selection interview. Outcome variables were graduation from the program in a timely fashion, or passing the final clinical skills assessment at the first attempt. Optimal selection cut-scores determined by discriminant function analysis for each selection tool at each school; efficacy of different selection algorithms for predicting student outcomes. For both outcomes, the cut-scores for prior academic achievement had the greatest predictive value, with medium to very large effect sizes (0.44-1.22) at all five schools. UMAT scores and selection interviews had smaller effect sizes (0.00-0.60). Meeting one or more cut-scores was associated with a significantly greater likelihood of timely graduation in some schools but not in others. An optimal cut-score can be estimated for a selection tool used for predicting an important program outcome. A "sufficient evidence" selection algorithm, founded on a non-compensatory model, is feasible, and may be useful for some schools.

The relative predictive value of undergraduate versus graduate selection tools in two Australian medical schools

Context: Monash University and the University of Western Australia admit both school-leavers and graduates into their Bachelor of Medicine and Bachelor of Surgery (MBBS) courses. The Undergraduate Medicine and Health Sciences Admission Test (UMAT) and the Graduate Medical Schools Admissions Test (GAMSAT) are used for selection, along with an academic score and an interview score. The aim of this study was to compare the relative predictive validity of the selected components in the two entry streams, particularly UMAT versus GAMSAT.Methods: Aggregated scores for course outcomes were calculated in the categories of knowledge, clinical and total scores, at four-time points. A path analysis was conducted based on multivariate regressions with model constraint parameters defined across the outcome variables to investigate change over time.Results: Academic scores were the strongest predictors of knowledge scores and end of course results. Interview scores had a small positive increasing effect, being stronger for clinical than knowledge outcomes. The effect size for GAMSAT was greater than for UMAT.Conclusions: Aptitude tests and interview scores added small but significant incremental predictive value to previous academic achievement. GAMSAT showed larger predictive value on outcomes than UMAT, for which one section (UMAT 3) had a negative effect.

Have we got the selection process right? The validity of selection tools for predicting academic performance in the first year of undergraduate medicine.

This article was migrated. The article was marked as recommended. Content: There remains much debate over the 'best' method for selecting students in to medicine. This study aimed to assess the predictive validity of four different selection tools with academic performance outcomes in first-year undergraduate medical students. Methods: Regression analyses were conducted between admission scores on previous academic performance - the Australian Tertiary Admission Rank (ATAR), the Undergraduate Medicine and Health Sciences Admission Test (UMAT), Multiple-Mini Interview (MMI) and the Personal Qualities Assessment (PQA) with student performance in first-year assessments of Multiple Choice Questions, Short Answer Questions, Objective Structured Clinical Examinations (OSCE) and Problem-Based Learning (PBL) Tutor ratings in four cohorts of students (N = 604, 90%). Results: All four selection tools were found to have significant predictive associations with one or more measures of student performance in Year One of undergraduate medicine. UMAT, ATAR and MMI scores consistently predicted first year performance on a number of outcomes. ATAR was the only selection tool to predict the likelihood of making satisfactory progress overall. Conclusions: All four selection tools play a contributing role in predicting academic performance in first year medical students. Further research is required to assess the validity of selection tools in predicting performance in the later years of medicine.

On the validity of repeated assessments in the UMAT, a high-stakes admissions test.

The possibility that the validity of assessment is compromised by repeated sittings of highly competitive and high profile selection tests has been documented and is of concern to stake-holders. An illustrative example is the Undergraduate Medicine and Health Sciences Admission Test (UMAT) used by some medical and dental courses in Australia and New Zealand. The proficiencies of all applicants who sat the UMAT from one to four sittings between 2006 and 2012 were estimated on the same metric using the probabilistic Rasch model. A fit index characterising each profile's degree of conformity to the model was also calculated. Confirming expectations, mean proficiencies increased with repeated sittings on all three UMAT scales with the greatest difference (which was nevertheless relatively small) between the first two sittings. The fit index showed that the increases in proficiency estimates arose from additional easier items being answered correctly on repeated sittings rather than additional more difficult ones, suggesting that improvements are not on the substantive construct of the variable of assessment but in skills in answering the questions. Although strategies for dealing with the increase in proficiency estimates on repeated sittings could be canvassed, these results suggest that the validity of results on repeated sittings was not compromised. Accordingly, it might be concluded that although particular individuals might improve substantially between sittings, any validity is not likely to be compromised with the possibility that for some applicants, the second sitting might be the most valid.

Selecting for a sustainable workforce to meet the future healthcare needs of rural communities in Australia.

An undersupply of generalists doctors in rural communities globally led to widening participation (WP) initiatives to increase the proportion of rural origin medical students. In 2002 the Australian Government mandated that 25% of commencing Australian medical students be of rural origin. Meeting this target has largely been achieved through reduced standards of entry for rural relative to urban applicants. This initiative is based on the assumption that rural origin students will succeed during training, and return to practice in rural locations. One aim of this study was to determine the relationships between student geographical origin (rural or urban), selection scores, and future practice intentions of medical students at course entry and course exit. Two multicentre databases containing selection and future practice preferences (location and specialisation) were combined (5862), representing 54% of undergraduate medical students commencing from 2006 to 2013 across nine Australian medical schools. A second aim was to determine course performance of rural origin students selected on lower scores than their urban peers. Selection and course performance data for rural (461) and urban (1431) origin students commencing 2006-2014 from one medical school was used. For Aim 1, a third (33.7%) of rural origin students indicated a preference for future rural practice at course exit, and even fewer (6.7%) urban origin students made this preference. Results from logistic regression analyses showed significant independent predictors were rural origin (OR 4.0), lower Australian Tertiary Admissions Rank (ATAR) (OR 2.1), or lower Undergraduate Medical and Health Sciences Admissions Test Section3 (non-verbal reasoning) (OR 1.3). Less than a fifth (17.6%) of rural origin students indicated a preference for future generalist practice at course exit. Significant predictors were female gender (OR 1.7) or lower ATAR (OR 1.2), but not rural origin. Fewer (10.5%) urban origin students indicated a preference for generalist practice at course exit. For Aim 2, results of Mann-Whitney U tests confirmed that slightly reducing selection scores does not result in increased failure, or meaningfully impaired performance during training relative to urban origin students. Our multicentre analysis supports success of the rural origin WP pathway to increase rural student participation in medical training. However, our findings confirm that current selection initiatives are insufficient to address the continuing problem of doctor maldistribution in Australia. We argue for further reform to current medical student selection, which remains largely determined by academic meritocracy. Our findings have relevance to the selection of students into health professions globally.

The interaction of socio-economic status and gender in widening participation in medicine.

The lack of representation of people from low socio-economic and socio-educational backgrounds in the medical profession is of growing concern and yet research investigating the problem typically studies recruitment and selection in isolation. This study examines the impacts of home and school socio-economic status (SES) from application to selection in an undergraduate medical degree. Socio-cognitive career theory and stereotype bias are used to explain why those from backgrounds of low SES may be disadvantaged, especially if they are female. Home and high school SES information for 2955 applicants and 202 medical students at one Australian medical school was related to application rates and performance on three selection tests (high school matriculation, the Undergraduate Medical and Health Sciences Admissions Test [UMAT] cognitive ability test, a multiple mini-interview) and academic performance in medical school. Interactions between gender and SES were assessed using moderated regression analyses. Applicants from backgrounds of low SES were under-represented. They were further disadvantaged at selection by the use of high school matriculation and cognitive ability tests, but not by the interview. They did not perform more poorly in medical school. Although females applied in greater numbers, a significant interaction between SES and gender indicated that female applicants of low SES were the most disadvantaged by the use of cognitive ability testing at selection. A targeted allowance of applicants from regions of low SES overcame this adverse impact to some extent. Efforts to widen participation that focus on recruitment are insufficient when selection tests have adverse impacts on people from backgrounds of low SES. The addressing of low self-efficacy that arises from socio-cultural factors, together with reductions in stereotype threat, may reduce the current disadvantages imposed by SES in the medical profession.

Medical student selection criteria as predictors of intended rural practice following graduation.

BackgroundRecruiting medical students from a rural background, together with offering them opportunities for prolonged immersion in rural clinical training environments, both lead to increased participation in the rural workforce after graduation. We have now assessed the extent to which medical students’ intentions to practice rurally may also be predicted by either medical school selection criteria and/or student socio-demographic profiles.MethodsThe study cohort included 538 secondary school-leaver entrants to The University of Western Australia Medical School from 2006 to 2011. On entry they completed a questionnaire indicating intention for either urban or rural practice following graduation. Selection factors (standardised interview score, percentile score from the Undergraduate Medicine and Health Sciences Admission Test (UMAT) and prior academic performance (Australian Tertiary Admissions Rank), together with socio-demographic factors (age, gender, decile for the Index of Relative Socioeconomic Advantage and Disadvantage (IRSAD) and an index of rurality) were examined in relation to intended rural or urban destination of practice.ResultsIn multivariate logistic regression, students from a rural background had a nearly 8-fold increase in the odds of intention to practice rurally after graduation compared to those from urban backgrounds (OR 7.84, 95% CI 4.10, 14.99, P < 0.001). Those intending to be generalists rather than specialists had a more than 4-fold increase in the odds of intention to practice rurally (OR 4.36, 95% CI 1.69, 11.22, P < 0.001). After controlling for these 2 factors, those with rural intent had significantly lower academic entry scores (P = 0.002) and marginally lower interview scores (P = 0.045). UMAT percentile scores were no different. Those intending to work in a rural location were also more likely to be female (OR 1.93, 95% CI 1.08, 3.48, P = 0.027), to come from the lower eight IRSAD deciles (OR 2.52, 95% CI 1.47, 4.32, P = 0.001) and to come from Government vs independent schools (OR 2.02, 95% CI 1.15, 3.55, P = 0.015).ConclusionsVery high academic scores generally required for medical school entry may have the unintended consequence of selecting fewer graduates interested in a rural practice destination. Increased efforts to recruit students from lower socioeconomic backgrounds may be beneficial in terms of an ultimate intended rural practice destination.Electronic supplementary materialThe online version of this article (doi:10.1186/1472-6920-14-218) contains supplementary material, which is available to authorized users.

Predictive validity of a new integrated selection process for medical school admission.

BackgroundThis paper is an evaluation of an integrated selection process utilising previous academic achievement [Universities Admission Index (UAI)], a skills test [Undergraduate Medicine and Health Sciences Admission Test (UMAT)], and a structured interview, introduced (in its entirety) in 2004 as part of curriculum reform of the undergraduate Medicine Program at the University of New South Wales (UNSW), Australia. Demographic measures of gender, country of birth, educational background and rurality are considered.MethodAdmission scores and program outcomes of 318 students enrolled in 2004 and 2005 were studied. Regression analyses were undertaken to determine whether selection scores predicted overall, knowledge-based and clinical-based learning outcomes after controlling for demographics.ResultsUAI attained the highest values in predicting overall and knowledge-based outcomes. The communication dimension of the interview achieved similar predictive values as UAI for clinical-based outcomes, although predictive values were relatively low. The UMAT did not predict any performance outcome. Female gender, European/European-derived country of birth and non-rurality were significant predictors independent of UAI scores.ConclusionResults indicate promising validity for an integrated selection process introduced for the Medicine Program at UNSW, with UAI and interview predictive of learning outcomes. Although not predictive, UMAT may have other useful roles in an integrated selection process. Further longitudinal research is proposed to monitor and improve the validity of the integrated student selection process.

Practice effects in medical school entrance testing with the undergraduate medicine and health sciences admission test (UMAT)

BackgroundThe UMAT is widely used for selection into undergraduate medical and dental courses in Australia and New Zealand (NZ). It tests aptitudes thought to be especially relevant to medical studies and consists of 3 sections – logical reasoning and problem solving (UMAT-1), understanding people (UMAT-2) and non-verbal reasoning (UMAT-3). A substantial proportion of all candidates re-sit the UMAT. Re-sitting raises the issue as to what might be the precise magnitude and determinants of any practice effects on the UMAT and their implications for equity in subsequent selection processes.MethodsBetween 2000 and 2012, 158,909 UMAT assessments were completed. From these, 135,833 cases were identified where a candidate had sat once or more during that period with 117,505 cases (86.5%) having sat once, 14,739 having sat twice (10.9%), 2,752 thrice (2%) and 837, 4 or more times (0.6%). Subsequent analyses determined predictors of multiple re-sits as well as the magnitude and socio-demographic determinants of any practice effects.ResultsIncreased likelihood of re-sitting the UMAT twice or more was predicted by being male, of younger age, being from a non-English language speaking background and being from NZ and for Australian candidates, being urban rather than rurally based. For those who sat at least twice, the total UMAT score between a first and second attempt improved by 10.7 ± 0.2 percentiles, UMAT-1 by 8.3 ± 0.2 percentiles, UMAT-2 by 8.3 ± 0.2 percentiles and UMAT-3 by 7.7 ± 0.2 percentiles. An increase in total UMAT percentile score on re-testing was predicted by a lower initial score and being a candidate from NZ rather than from Australia while a decrease was related to increased length of time since initially sitting the test, older age and non-English language background.ConclusionsRe-sitting the UMAT augments performance in each of its components together with the total UMAT percentile score. Whether this increase represents just an improvement in performance or an improvement in understanding of the variables and therefore competence needs to be further defined. If only the former, then practice effects may be introducing inequity in student selection for medical or dental schools in Australia or NZ.

The impact of preparatory activities on medical school selection outcomes: a cross-sectional survey of applicants to the university of Adelaide medical school in 2007

BackgroundSelection into medical school is highly competitive with more applicants than places. Little is known about the preparation that applicants undertake for this high stakes process. The study aims to determine what preparatory activities applicants undertake and what difficulties they encounter for each stage of the application process to medical school and in particular what impact these have on the outcome.MethodsA cross-sectional survey of 1097 applicants who applied for a place in the University of Adelaide Medical School in 2007 and participated in the UMAT (Undergraduate Medicine and Health Sciences Admission Test) and oral assessment components of the selection process. The main outcome measures were an offer of an interview and offer of a place in the medical school and were analysed using logistic regression.ResultsThe odds of a successful outcome increased with each additional preparatory activity undertaken for the UMAT (odds ratio 1.22, 95% confidence interval 1.11 to 1.33; P < 0.001) and the oral assessment (1.36, 1.19 to 1.55; P < 0.001) stage of selection. The UMAT preparatory activities associated with the offer of an interview were attendance of a training course by a private organisation (1.75, 1.35 to 2.27: P < 0.001), use of online services of a private organisation (1.58, 1.23 to 2.04; P < 0.001), and familiarising oneself with the process (1.52, 1.15 to 2.00; p = 0.021). The oral assessment activities associated with an offer of a place included refining and learning a personal resume (9.73, 2.97 to 31.88; P < 0.001) and learning about the course structure (2.05, 1.29 to 3.26; P = 0.022).For the UMAT, applicants who found difficulties with learning for this type of test (0.47, 0.35 to 0.63: P < 0.001), with the timing of UMAT in terms of school exams (0.48, 0.5 to 0.66; P < 0.001) and with the inability to convey personal skills with the UMAT (0.67, 0.52 to 0.86; P = 0.026) were significantly less likely to be offered an interview.ConclusionsMedical schools make an enormous effort to undertake a selection process that is fair and equitable and which selects students most appropriate for medical school and the course they provide. Our results indicate that performance in the selection processes can be improved by training. However, if these preparatory activities may be limited to those who can access them, the playing field is not even and increasing equity of access to medical schools will not be achieved.

Socio-economic predictors of performance in the Undergraduate Medicine and Health Sciences Admission Test (UMAT)

BackgroundEntry from secondary school to Australian and New Zealand undergraduate medical schools has since the late 1990’s increasingly relied on the Undergraduate Medicine and Health Sciences Admission Test (UMAT) as one of the selection factors. The UMAT consists of 3 sections – logical reasoning and problem solving (UMAT-1), understanding people (UMAT-2) and non-verbal reasoning (UMAT-3). One of the goals of using this test has been to enhance equity in the selection of students with the anticipation of an increase in the socioeconomic diversity in student cohorts. However there has been limited assessment as to whether UMAT performance itself might be influenced by socioeconomic background.MethodsBetween 2000 and 2012, 158,909 UMAT assessments were completed. From these, 118,085 cases have been identified where an Australian candidate was sitting for the first time during that period. Predictors of the total UMAT score, UMAT-1, UMAT-2 and UMAT-3 scores were entered into regression models and included gender, age, school type, language used at home, deciles for the Index of Relative Socioeconomic Advantage and Disadvantage score, the Accessibility/Remoteness Index of Australia (ARIA), self-identification as being of Aboriginal or Torres Strait Islander origin (ATSI) and current Australian state or territory of abode.ResultsA lower UMAT score was predicted by living in an area of relatively higher social disadvantage and lower social advantage. Other socioeconomic indicators were consistent with this observation with lower scores in those who self-identified as being of ATSI origin and higher scores evident in those from fee-paying independent school backgrounds compared to government schools. Lower scores were seen with increasing age, female gender and speaking any language other than English at home. Divergent effects of rurality were observed, with increased scores for UMAT-1 and UMAT-2, but decreasing UMAT-3 scores with increasing ARIA score. Significant state-based differences largely reflected substantial socio-demographic differences across Australian states and territories.ConclusionsBetter performance by Australian candidates in the UMAT is linked to an increase in socio-economic advantage and reduced disadvantage.This observation provides a firm foundation for selection processes at medical schools in Australia that have incorporated affirmative action pathways to quarantine places for students from areas of socio-economic disadvantage.

Students coached for an admission test perform less well throughout a medical course

Undertaking commercial coaching to improve one's chance of selection into medical school is widespread. Although its effect on selection test performance appears to be relatively minimal, its impact on the predictive validity of the tests is unknown. To examine whether commercial coaching for the Undergraduate Medical and Health Sciences Admissions Test (UMAT) changes its ability to predict the subsequent academic performance of medical students. The first two cohorts to enrol in a new Australian medical school provided information at the time of their selection interview about whether or not they had undertaken a commercial coaching course to help prepare for the UMAT. Final academic grades for each year of the degree and overall grade point average (GPA) of coached students were compared with those of non-coached students. Moderated regression analyses examined differences in the relationship between UMAT scores and examination results while controlling for entry UMAT scores and past academic performance. Coached students had a lower GPA than those who were not coached. In cohort 1, coached students performed more poorly than non-coached students in every year of their degree. This effect, while similar, was not statistically significant in cohort 2. Differences in selection process and learning context between the two cohorts may explain why coaching was only significantly related to the performance of one cohort. Further research is required to ascertain if coached students develop a learning style that hinders ongoing acquisition of knowledge, which might have serious implications for job performance after graduation.

The Undergraduate Medical and Health Sciences Admissions Test: What is it measuring?

Background: The Undergraduate Medical and Health Sciences Admissions Test (UMAT) is used to select medical students in Australia and New Zealand but empirical evidence of its construct validity has never been reported.Aims: To identify the underlying constructs assessed in each of the three sections of the UMAT. Based on conclusions from an early qualitative study (Mercer & Chiavaroli 2006), it was expected that Section 1 scores would correlate with scores obtained from standard measures of cognitive ability (verbal and numeric reasoning), Section 2 scores would correlate with emotional intelligence, and Section 3 scores would be most strongly related to abstract or non-verbal reasoning ability.Method: Final year high school students (n = 432) completed tests of numerical, verbal, and non-verbal cognitive ability, and emotional intelligence. Correlations and multiple regressions assessed the relationship of these tests with scores on each section of the UMAT.Results: UMAT Section 1 was significantly related to verbal, non-verbal, and numerical reasoning tests. Section 2 was significantly related to emotional intelligence and verbal reasoning, but the majority of variance in this section's scores remained unexplained. Section 3 scores significantly correlated with non-verbal and numerical reasoning.Conclusions: The UMAT Sections 1 and 3 appear to be tests of cognitive abilities. Further research is required to identify the constructs being measured by Section 2.

To what extent does the Health Professions Admission Test-Ireland predict performance in early undergraduate tests of communication and clinical skills? An observational cohort study.

BackgroundInternationally, tests of general mental ability are used in the selection of medical students. Examples include the Medical College Admission Test, Undergraduate Medicine and Health Sciences Admission Test and the UK Clinical Aptitude Test. The most widely used measure of their efficacy is predictive validity.A new tool, the Health Professions Admission Test- Ireland (HPAT-Ireland), was introduced in 2009. Traditionally, selection to Irish undergraduate medical schools relied on academic achievement. Since 2009, Irish and EU applicants are selected on a combination of their secondary school academic record (measured predominately by the Leaving Certificate Examination) and HPAT-Ireland score. This is the first study to report on the predictive validity of the HPAT-Ireland for early undergraduate assessments of communication and clinical skills.MethodStudents enrolled at two Irish medical schools in 2009 were followed up for two years. Data collected were gender, HPAT-Ireland total and subsection scores; Leaving Certificate Examination plus HPAT-Ireland combined score, Year 1 Objective Structured Clinical Examination (OSCE) scores (Total score, communication and clinical subtest scores), Year 1 Multiple Choice Questions and Year 2 OSCE and subset scores. We report descriptive statistics, Pearson correlation coefficients and Multiple linear regression models.ResultsData were available for 312 students. In Year 1 none of the selection criteria were significantly related to student OSCE performance. The Leaving Certificate Examination and Leaving Certificate plus HPAT-Ireland combined scores correlated with MCQ marks.In Year 2 a series of significant correlations emerged between the HPAT-Ireland and subsections thereof with OSCE Communication Z-scores; OSCE Clinical Z-scores; and Total OSCE Z-scores. However on multiple regression only the relationship between Total OSCE Score and the Total HPAT-Ireland score remained significant; albeit the predictive power was modest.ConclusionWe found that none of our selection criteria strongly predict clinical and communication skills. The HPAT- Ireland appears to measures ability in domains different to those assessed by the Leaving Certificate Examination. While some significant associations did emerge in Year 2 between HPAT Ireland and total OSCE scores further evaluation is required to establish if this pattern continues during the senior years of the medical course.

Preparation courses for a medical admissions test: effectiveness contrasts with opinion

The Undergraduate Medicine and Health Sciences Admissions Test (UMAT) is used to rank applicants for admission to many medical schools. This study aimed to determine the effects of preparation courses on UMAT performance and on students' perceptions of their performance. We asked students who sat the UMAT twice across two consecutive years to complete an online survey. The survey was administered in 2010 and 2011 to gather information on preparation activities, costs of preparation activities and students' opinions regarding their expected performance. Survey responses were compared with student scores on the second taking of the UMAT, adjusted for prior UMAT scores and university performance. The study (cohort: n = 263) was sufficiently powered to investigate two forms of preparation: courses offered by MedEntry (a UMAT preparation provider), and tutoring offered by the students' halls of residence. Neither was found to significantly affect UMAT score (p = 0.09 for MedEntry courses; p = 0.50 for halls of residence tutoring). There was no relationship between total time or money spent preparing and UMAT performance. However, students taking MedEntry courses and students spending more money on UMAT preparation were more likely to predict an improved score (p < 0.001 for both). A total of 85% of students improved their score on the second sitting, irrespective of preparation. The use of either of two common forms of UMAT preparation does not translate to an improvement in UMAT score. However, in accordance with confirmation bias theory, the association between money spent on preparatory courses and self-assessed predicted score suggests that students' belief in the effectiveness of such courses may be confounded by their financial outlay.