Abstract
Combined with neural language models, distributed word representations achieve significant advantages in computational linguistics and text mining. Most existing models estimate distributed word vectors from large-scale data in an unsupervised fashion, which, however, do not take rich linguistic knowledge into consideration. Linguistic knowledge can be represented as either link-based knowledge or preference-based knowledge, and we propose knowledge regularized word representation models (KRWR) to incorporate these prior knowledge for learning distributed word representations. Experiment results demonstrate that our estimated word representation achieves better performance in task of semantic relatedness ranking. This indicates that our methods can efficiently encode both prior knowledge from knowledge bases and statistical knowledge from large-scale text corpora into a unified word representation model, which will benefit many tasks in text mining.
Highlights
The performance of text mining is heavily dependent on word representation
JO-SPR indicates Softmax Probability Regularizer trained by Joint Optimization, PO-SPR means Softmax Probability Regularizer trained by Post Optimization, and PO-ER means Euclidean Regularizer trained by Post Optimization
We propose a unified framework to incorporate prior knowledge into distributed word representation
Summary
The most widely used methods of word representation are vector space models (VSM) [1], which represent word meanings with vectors, with each dimension corresponding to semantic or syntactic information of words. VSM can be used to conduct similarity measures by computing distances between vectors, and are widely adopted in various applications such as information retrieval, text classification and question answering. It has long been known that simple co-occurrence counts do not work well for DSM. Techniques such as reweighting, smoothing and dimension reduction have been proposed to enhance performance [2]. These optimization techniques require heavily manual tuning. DSM is non-trivial to be extended to higher level representation of sentences or documents
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
