Survey of BERT (Bidirectional Encoder Representation Transformer) types

Abstract

There are many algorithms used in Natural Language Processing( NLP) to achieve good results, such as Machine Learning (ML), Deep Learning(DL) and many other algorithms. In Natural Language Processing,the first challenges is to convert text to numbers for using by any algorithm that a researcher choose. So how can convert text to numbers? This is happen by using Word Embedding algorithms such as skip gram,bags of words,BERT and etc. Representing words as numerical vectors by relying on the contents has become one of the effective methods for analyzing texts in machine learning, so that each word is represented by a vector to determine its meaning or to know how close or distant this word from the rest of the other word. BERT(Bidirectional Encoder Representation Transformer) is one of the embedding methods. It is designed to pre-trained form left and right in all layer deep training. It is a deep language model that is used for various tasks in natural language processing. In this paper we will review the different versions and types of BERT.