KCC QA Latent Semantic Representation using Deep Learning & Hierarchical Semantic cluster Inferential Framework

Abstract

Abstract Rapid digitization is in progress in even the most traditional domains like Agriculture. In line with this, the Government of India released call centre question-answer data called the Kisan Call Centre data ( KCC ). Using the KCC data available at Open Government Data platform this paper suggests a novel double-headed autoencoder architecture that outperforms ubiquitous deep learning architectures used as baselines. Experiments are carried out for the data from Tamil Nadu call centres. The various sentence embeddings generated are clustered and externally analyzed against carefully hand-annotated data using various measures like cluster entropy for various levels of semantic resolution. A framework is created to model and evaluate semantic embeddings to create scope for various downstream tasks such as predicting rare events like drought and low yield etc