Abstract
Tokenization of modern and old Western European languages seems to be fairly simple, as it stands on the presence mostly of markers such as spaces and punctuation. However, when dealing with old sources like manuscripts written in scripta continua, antiquity epigraphy or Middle Age manuscripts, (1) such markers are mostly absent, (2) spelling variation and rich morphology make dictionary based approaches difficult. Applying convolutional encoding to characters followed by linear categorization to word-boundary or in-word-sequence is shown to be effective at tokenizing such inputs. Additionally, the software is released with a simple interface for tokenizing a corpus or generating a training set.
Highlights
Tokenization of spaceless strings is a task that is difficult for computers as compared to ”whathumanscando”
In the context of text mining of HTR or OCR output, lemmatization and tokenization of medieval western languages is quite often a pre-processing step for further research to sustain analyses such as authorship attribution, corpus linguistics or to allow full-text search 3. It must be stressed in this study that the difficulty inherent to segmentation is different for scripta continua than the one for languages such as Chinese, for which an already impressive amount of work has been done
Output of the model is a mask that needs to be applied to the input: in the mask, characters are classified either as word boundary or word content
Summary
Tokenization of spaceless strings is a task that is difficult for computers as compared to ”whathumanscando”. In the context of text mining of HTR or OCR output, lemmatization and tokenization of medieval western languages is quite often a pre-processing step for further research to sustain analyses such as authorship attribution, corpus linguistics or to allow full-text search 3. It must be stressed in this study that the difficulty inherent to segmentation is different for scripta continua than the one for languages such as Chinese, for which an already impressive amount of work has been done. This makes a dictionary-based approach rather difficult as it would rely on a high number of different spellings, making the computation highly complex
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have