Actual Information Need Research Articles

Multi-Task Learning for Entity Recommendation and Document Ranking in Web Search

Entity recommendation, providing users with an improved search experience by proactively recommending related entities to a given query, has become an indispensable feature of today’s Web search engine. Existing studies typically only consider the query issued at the current timestep while ignoring the in-session user search behavior (short-term search history) or historical user search behavior across all sessions (long-term search history) when generating entity recommendations. As a consequence, they may fail to recommend entities of interest relevant to a user’s actual information need. In this work, we believe that both short-term and long-term search history convey valuable evidence that could help understand the user’s search intent behind a query, and take both of them into consideration for entity recommendation. Furthermore, there has been little work on exploring whether the use of other companion tasks in Web search such as document ranking as auxiliary tasks could improve the performance of entity recommendation. To this end, we propose a multi-task learning framework with deep neural networks (DNNs) to jointly learn and optimize two companion tasks in Web search engines: entity recommendation and document ranking, which can be easily trained in an end-to-end manner. Specifically, we regard document ranking as an auxiliary task to improve the main task of entity recommendation, where the representations of queries, sessions, and users are shared across all tasks and optimized by the multi-task objective during training. We evaluate our approach using large-scale, real-world search logs of a widely-used commercial Web search engine. We also performed extensive ablation experiments over a number of facets of the proposed multi-task DNN model to figure out their relative importance. The experimental results show that both short-term and long-term search history can bring significant improvements in recommendation effectiveness, and the combination of both outperforms using either of them individually. In addition, the experiments show that the performance of both entity recommendation and document ranking can be significantly improved, which demonstrates the effectiveness of using multi-task learning to jointly optimize the two companion tasks in Web search.

Secret Shopping is an Effective Tool for Identifying Local Patterns in Library User Experience

A Review of:
 Boyce, C. M. (2015). Secret shopping as user experience assessment tool. Public Services Quarterly, 11(4), 237-253. doi:10.1080/15228959.2015.1084903
 
 Objective – To assess library user experience (UX) at two entry-level service desks to determine the need for, and inform the aspects in which to improve, services and staff training. 
 
 Design – Observational study using secret shopping.
 
 Setting – A small, private university in Illinois, United States of America.
 
 Subjects – Library employees, comprised primarily of student assistants; and 11 secret shoppers, comprised of 5 faculty members, 4 staff members, and 2 first-year students from the university. 
 
 Methods – Recruitment methods for shoppers consisted of the campus electronic mail list, flyers, directed requests, and a $10 gift certificate to the campus bookstore following their participation. Both groups (library employees and secret shoppers) were briefed on the purpose of the study and completed informed consent forms. Shoppers attended face-to-face training sessions in which they selected two questions to ask from a list of options, one for each service desk in the library. Shoppers were not told at which desk to ask their questions. The list of options included informational and research assistance questions; shoppers also had the option of asking a question based on an actual information need. Two service desks were involved: one for circulation and one for research/IT support. Since IT staff and Librarians were excluded from the study, shoppers were directed to tactfully end the transaction if a referral to an expert was made. 
 
 Within two weeks of the training session, shoppers made two separate visits to the library at a time convenient to them to ask the question and observe the transaction at each of the two service desks. For each round of secret shopping, shoppers completed an electronic evaluation form afterward on the Qualtrics platform. The evaluation form consisted of yes/no, multiple-choice, and open-ended comment questions with two questions employing skip logic for a total of 29 possible questions. The questions covered the following variables both quantitatively and qualitatively: how well the question was answered, the customer service skills (responsiveness, approachability, and respectfulness) of the library employee, and if applicable, the quality of the referral to other library staff or services. 
 
 Main Results – The shoppers evaluated a total of 21 transactions: 11 for the circulation desk and 10 for the research/IT support desk (1 shopper did not evaluate this desk). Eighteen of the transactions were in-person and three3 were by phone. Eight of the questions asked were based on the participants’ actual information need.
 
 On the variable of satisfaction with the answer received, the research/IT support desk scored higher than the circulation desk. The circulation desk received 7 very satisfactory ratings, 3 satisfied ratings, and 1 neutral rating; whereas the research/IT support desk received 10 very satisfactory ratings and 1 satisfied rating. The lower scores of the circulation desk may be related to the variables of responsiveness and approachability, as library employees on the circulation desk were scored lower in these areas and observed as not paying attention in two interactions and this was not observed at all at the research/IT support desk. However, the study did not collect sufficient data to test this relationship. All shoppers gave positive ratings on whether they were treated respectfully and if the library employee waited for the shopper to state the question fully and with the exception of one transaction, clarify the question if necessary. 
 Responses to the open-ended comment questions were reviewed by investigators, who found that in five cases the transaction would have been improved by consulting library faculty. On the variables related to customer service, responses were generally positive but in several transactions the library employees failed to appear attentive and ready to help the shopper. 
 
 Conclusion – The author found secret shopping was an effective tool for evaluating library UX to identify both positive and negative patterns and better inform responses to areas in need of improvement. The author identified two areas for improvement to the library. First, library employees at the circulation desk require additional training that would encourage them to refer transactions to library faculty where necessary. Second, although evaluation of customer service skills was generally positive, library employees will also receive additional training that will emphasize listening and role-playing scenarios. These areas for improvement will also support the library’s plans to combine research and circulation functions into a single service desk on the entry level and move the IT support desk to the third floor.

Generating query suggestions by exploiting latent semantics in query logs

Search engines assist users in expressing their information needs more accurately by reformulating the issued queries automatically and suggesting the generated formulations to the users. Many approaches to query suggestion draw on the information stored in query logs, recommending recorded queries that are textually similar to the current user’s query or that frequently co-occurred with it in the past. In this paper, we propose an approach that concentrates on deducing the actual information need from the user’s query. The challenge therein lies not only in processing keyword queries, which are often short and possibly ambiguous, but especially in handling the complexity of natural language that allows users to express the same or similar information needs in various differing ways. We expect a higher-level semantic representation of a user’s query to more accurately reflect the information need than the explicit query terms alone can. To this aim, we employ latent Dirichlet allocation as a probabilistic topic model to reveal latent semantics in the query log. Our evaluations show that, whereas purely topic-based query suggestion performs the worst, the interpolation of our proposed topic-based model with the baseline word-based model that generates suggestions based on matching query terms achieves significant improvements in suggestion quality over the already well performing purely word-based approach.

Explicit web search result diversification

Queries submitted to a web search engine are typically short and often ambiguous. With the enormous size of the Web, a misunderstanding of the information need underlying an ambiguous query can misguide the search engine, ultimately leading the user to abandon the originally submitted query. In order to overcome this problem, a sensible approach is to diversify the documents retrieved for the user's query. As a result, the likelihood that at least one of these documents will satisfy the user's actual information need is increased. In this thesis, we argue that an ambiguous query should be seen as representing not one, but multiple information needs. Based upon this premise, we propose xQuAD---Explicit Query Aspect Diversification, a novel probabilistic framework for search result diversification. In particular, the xQuAD framework naturally models several dimensions of the search result diversification problem in a principled yet practical manner. To this end, the framework represents the possible information needs underlying a query as a set of keyword-based sub-queries. Moreover, xQuAD accounts for the overall coverage of each retrieved document with respect to the identified sub-queries, so as to rank highly diverse documents first. In addition, it accounts for how well each sub-query is covered by the other retrieved documents, so as to promote novelty--and hence penalise redundancy---in the ranking. The framework also models the importance of each of the identified sub-queries, so as to appropriately cater for the interests of the user population when diversifying the retrieved documents. Finally, since not all queries are equally ambiguous, the xQuAD framework caters for the ambiguity level of different queries, so as to appropriately trade-off relevance for diversity on a per-query basis. The xQuAD framework is general and can be used to instantiate several diversification models, including the most prominent models described in the literature. In particular, within xQuAD, each of the aforementioned dimensions of the search result diversification problem can be tackled in a variety of ways. In this thesis, as additional contributions besides the xQuAD framework, we introduce novel machine learning approaches for addressing each of these dimensions. These include a learning to rank approach for identifying effective subqueries as query suggestions mined from a query log, an intent-aware approach for choosing the ranking models most likely to be effective for estimating the coverage and novelty of multiple documents with respect to a sub-query, and a selective approach for automatically predicting how much to diversify the documents retrieved for each individual query. In addition, we perform the first empirical analysis of the role of novelty as a diversification strategy for web search. As demonstrated throughout this thesis, the principles underlying the xQuAD framework are general, sound, and effective. In particular, to validate the contributions of this thesis, we thoroughly assess the effectiveness of xQuAD under the standard experimentation paradigm provided by the diversity task of the TREC 2009, 2010, and 2011 Web tracks. The results of this investigation demonstrate the effectiveness of our proposed framework. Indeed, xQuAD attains consistent and significant improvements in comparison to the most effective diversification approaches in the literature, and across a range of experimental conditions, comprising multiple input rankings, multiple sub-query generation and coverage estimation mechanisms, as well as queries with multiple levels of ambiguity. Altogether, these results corroborate the state-of-the-art diversification performance of xQuAD. Available online at http://theses.gla.ac.uk/4106/.

How do users describe their information need: Query recommendation based on snippet click model

Query recommendation helps users to describe their information needs more clearly so that search engines can return appropriate answers and meet their needs. State-of-the-art researches prove that the use of users’ behavior information helps to improve query recommendation performance. Instead of finding the most similar terms previous users queried, we focus on how to detect users’ actual information need based on their search behaviors. The key idea of this paper is that although the clicked documents are not always relevant to users’ queries, the snippets which lead them to the click most probably meet their information needs. Based on analysis into large-scale practical search behavior log data, two snippet click behavior models are constructed and corresponding query recommendation algorithms are proposed. Experimental results based on two widely-used commercial search engines’ click-through data prove that the proposed algorithms outperform practical recommendation methods of these two search engines. To the best of our knowledge, this is the first time that snippet click models are proposed for query recommendation task.

Comparison between Ad-hoc Retrieval and Filtering Retrieval Using Arabic Documents

The objective of this research is to study the process of examining documents by computing comparisons between the representation of the information need (the queries) and the representations of the documents. Also, we will automate the process of representing information needs as user profiles by computing the comparison between the user profile and the representations of the documents. We consider an automated process to be successful when it produces results similar to those produced by human comparison of the documents themselves with actual information need. Thus, we will compare ad-hoc retrieval and filtering retrieval tasks and examine the differences between them in terms of the information retrieval process. We have selected 242 Arabic abstracts that were used by Hmeidi [7]. All these abstracts involve computer science and information systems. We have also designed and built a system to compare two different retrieval tasks: ad-hoc retrieval and filtering retrieval. Here, we define ad-hoc and filtering retrieval systems and illustrate the development strategy for each one. We compare the two tasks on the basis of recall/precision evaluation, system usability, domain search, ranking, construction complexity, and methodology. From this experiment, we conclude that ad-hoc retrieval gives better performance than filtering retrieval. We also consider the advantages of using filtering services in the information retrieval process.

ランカスターのメドラース実績評価を通してみた情報システムと利用者間の相互作用

The mode of literature searching currently used almost exclusively by large mechanized retrieval systems is one-chance delegated search. In this type of search, the information need is delegated by the person having the information need to a second individual who prepares searching strategy and formulation, and a number of searches are batched and processd simultaneously. The searcher has little opportunity to browse on behalf of the requester as he does in the conventional manual literature searching. It is obvious that with this type of searching mode a very clear and detailed statement of the requester’s information need is prerequisite to the successful search. Then, it leads to assume that the interface between the user and the system will importantly affect the performance of a large mechanized retrieval system. The present paper is concerned with question of user-system interaction of a large mechanized retrieval system as observed in the evaluation study of MEDLARS conducted at the National Library of Medicine of the United States. The significance of user-system interaction is well illustrated by the results of the study. It is indicated that 25% of all the recall failures and 16.6% of all the precision failures are attributed to undesirable interaction between the user and the system. A recall failure due to defective interaction is mainly attributable to the system. A recall failure due to defective interaction is mainly attributable to the request statement which is more specific than the actual area of information need. On the other hand, a precision failure is due to a request statement which is more general than the actual information need. The importance of a request statement that explicitly delineates the actual information need is sufficiently emphasized.