"What's this?": Understanding User Interaction Behaviour with Multimodal Input Information Retrieval System

Abstract

Human communication relies on integrated multimodal channels to facilitate rich information exchange. Building on this foundation, researchers have long speculated about the potential benefits of incorporating multimodal input channels into conventional information retrieval (IR) systems to support users’ complex daily IR tasks more effectively. However, the true benefits of such integration remain uncertain. This paper presents a series of exploratory pilot tests comparing Multimodal Input IR (MIIR) with Unimodal Input IR (UIIR) across various IR scenarios, concluding that MIIR offers distinct advantages over UIIR in terms of user experiences.

Daniel (Danny) Hyeongcheol Kim

Human-Centered AI
Researcher & Builder

HCAI researcher and builder, Postdoctoral Research Fellow at KAIST and co-founder of KEONIX Labs.