On the evaluation of conversational search - Liu, Zeyang (2024)
The rapid growth of speech technology has significantly influenced the way that users interact with search systems to satisfy their information needs. Unlike the mode of textual interaction, Voice User Interfaces (VUIs) often encourage multiple rounds of interaction for reasons such as clarifying user input, error handling, progressive information disclosure, personalization, and completing complex tasks. On the other hand, VUIs also promote natural interaction by ensuring accurate speech recognition, maintaining context throughout conversations, offering clear prompts and guidance, and providing feedback and confirmation. Meanwhile, with the widespread application of large language models (LLMs), information retrieval systems increasingly engage in dialogue with users, such as Bing Chat and Google Bard. In comparison to the traditional query-document paradigm, conversational search systems often encourage users to express their search tasks using natural language and to interact over multiple rounds. By adopting this dialogue-like form, these systems can improve accuracy and better understand user intent. However, it is worth noting that the essence of a conversational search system is still an information retrieval (IR) system, which aims to provide users with the information that they want in order to satisfy their information needs. This interaction paradigm is more natural for humans and can help the users to better articulate their information needs. In turn, however, this search paradigm increases the complexity for systems to understand users’ intent and underlying information needs across the multi-turn interactions. Further, the added complexities mean that the evaluation of conversational search systems is also not straightforward. One of the challenges in evaluating conversational search systems is that the number of possible user utterances and system responses is infinite, thus it is difficult, if not impossible, to use a static, finite set of relevance labels to evaluate their effectiveness. Therefore, the evaluation of a conversational search system remains an open, non-trivial challenge. This thesis aims to address some aspects of this challenge and make contributions towards building a comprehensive and replicable evaluation framework for conversational search.