Smart question answering system using vectorization approach and statistical scoring method

Abstract

In this digital era, internet users are increasing day by day to extract information related to various fields such as health, economic, educational, politics and daily needs from Information Retrieval Systems (IRS) such as Google, Ask, Yahoo etc. Information Retrieval (IR) is a conventional method which depends on metadata searching method that is searching of adocuments for information within documents and information about documents. Lot of research in the IR area improves accessing the quality of information. Most of the search engines such as Google, Yahoo, Ask Jeeves depends on keyword based query processing mechanism internally which can’t handle natural language questions which results in returning long list of documents for a query. The task of users is to manually explore each document to retrieve correct answer which is a time consuming process. To overcome the problems of conventional search engines, smart question answering system comes to play. The objective of Smart Automated Answering system is to impart accurate and valid answers to user question in natural language rather than set of full documents.The Smart Question answering system which accepts user question in Natural language in specific domain and provide short and precise answer using Natural language processing technique, used for pre-processing, normalized term weighting-TF-IDF with cosine similarity approach for document retrieval and enhanced BM25 ranking function to extract answer. The main focus is on decreasing the response time, designing ranking function and giving the pertinent responses to the users’ intent. The system achieved an overall Precision of 93.2%, Recall of 84.3% and F-measure of 88.5% with 80% of Accuracy.