The Effect of Synonym Substitution on Search Results

Abstract

Synonyms or other semantic associations can be used in web search in query substitution to improve or augment the query to retrieve more relevant search results. The value of substitution depends on how well the synonyms preserve semantic meaning, as any attrition in meaning can result in semantic drift of query results. Many synonyms are not synonyms in the traditional, thesaurus sense, but are semantic associations discovered automatically from online data, with the risk of semantic drift in substitution. This discovery of synonyms or other semantic associations arises from different methods applied over web search logs, and in this paper we review the candidate synonym pairs of words or phrases generated from three different methods applied over the same web search logs. The suitability of the candidate synonym pairs for the purpose of query substitution is evaluated in an experiment where 68 subjects assessed the search results generated by both the original query and the substituted query. It was found that two of the discovery methods returned significantly worse results with the substitution than were returned by the original query for the majority of queries, with only around 20-22% of substituted queries generating either improved or equally-relevant results. The third method however returned a very similar level of superior results as the original query, and saw over 71% of substituted queries generating either improved or equally-relevant results. These results indicate that even when candidate synonym pairs are confirmed as being semantically associated using other methods, they still may not be suitable for query substitution, depending on the method of synonym discovery.