Abstract
Artificial intelligence is revolutionizing various work sectors, including human resource
 management, With the advent of ChatGPT in particular, AI has moved beyond simple task
 automation to potentially handle complex duties in HR. Due to the lack of specific studies
 on ChatGPT's performance, this research focuses on analyzing its efficacy in cover letter
 evaluation. ChatGPT was used to evaluate 23,940 units derived from 84 cover letters and
 evaluation sheets, with the results analyzed and compared to human evaluations. We found
 that an increase in the temperature, ChatGPT's key hyperparameter, led to rising error rates
 and declining rating consistency. We also found that ChatGPT tended to assign a bias
 toward certain ratings rather than assigning a variety of ratings evenly, with diversity
 levels influenced by the rating nomenclature used. Next, we identified significant positive
 correlations between ChatGPT and human raters, reaching up to 0.34 (p<.01) and 0.60
 (p<.001) for GPT-3.5 and GPT-4 models, respectively. We also found that ChatGPT's
 top-scoring applicants matched actual hires by 61% for the GPT-3.5 model and 83% for
 the GPT-4 model. These results suggest that ChatGPT could support or automate cover
 letter assessments. ChatGPT's potential was found to be significant, given its ability to
 evaluate a cover letter in under a minute for just $0.03. Based on these findings, the
 conclusion discusses the theoretical and practical implications of the study and directions
 for future research.
Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
