A natural language processing approach to skill identification in the job market

Abstract

The dynamic nature of the job market, with its constant growth and evolving demands, makes the assessment of skills needed for a particular job increasingly difficult, time-consuming, and tedious. This necessitates an automated approach to identify the required skills for job seekers and recruiters. This study aimed to utilize machine learning (ML) and deep learning algorithms to streamline skill identification from technology-related job descriptions. We tested the viability of the Bidirectional Encoder Representation of Transformers (BERT) language model to complete this multi-class, multi-label classification task. We hypothesized that BERT, utilizing an NLP transformer, would predict job-required skills more accurately than traditional statistical ML models. To test this, we utilized a dataset comprised of job descriptions with varying skills. We experimented with two preprocessing approaches, K-Means Clustering and Latent Dirichlet Allocation, which are both statistical ML models meant to cluster the job descriptions. We then compared the performance of the BERT tokenizer to a combination of text-to-number tokenization techniques (Term Frequency - Inverse Document Frequency and Word2Vec) and supervised classifiers (K-Nearest Neighbors, Random Forest, and Multi-Layer Perceptron). The NLP transformer-based BERT model achieved the highest accuracy. We conclude that while traditional ML models provide some insights, advanced models like BERT show potential for more accurate skill prediction since the NLP transformer model we tested outperformed the ML models. However, the complexity of real-world data necessitates further refinement and development of these technologies. Our study shows that advanced deep learning-based models have the potential to enhance the job market’s efficiency by automating skill identification.