RESIRE - Reproducibility and simulation of interactive retrieval experiments

Profile & Description

Although reproducibility of experiments is a topic with increasing importance in information retrieval (IR), there has been hardly any work on reproducing interactive IR experiments. The main idea of our proposed project is to describe actual users by quantitative modeling of their behavior and their content-related decisions during retrieval sessions. These model parameters can then form the basis for comparing two interactive IR experiments with regard to reproducibility, and can also be used for simulating interactive IR, e. g. for evaluating new systems or system variants without (or before) carrying out actual user experiments. As part of the proposed project, we will define a new framework model for reproducibility of interactive IR experiments, develop new metrics for reproducibility and session quality, design new simulation methods integrating behavior and content-related decisions, develop an open source framework for reproduction and simulation of interactive IR, and perform extensive experiments using existing IR log data as well as observation data from new user experiments.

Funding Agency
DFG - Deutsche Forschungsgemeinschaft
Partner Institution
University of Duisburg-Essen
People Involved
Prof. Dr. Philipp Schaer (Technische Hochschule Köln)
Dr. Timo Breuer (Technische Hochschule Köln)

ProjectRESIRE - Reproducibility and simulation of interactive retrieval experiments

Duration
2023 - 2025
Funded by

Publications

2024

Validating Synthetic Usage Data in Living Lab Environments.
Journal of Data and Information Quality, 16(1):1-33, 2024.
Timo Breuer, Norbert Fuhr and Philipp Schaer.
[doi] [pdf]  [BibTeX] 
Context-Driven Interactive Query Simulations Based on Generative Large Language Models.
In: ECIR 2024. 2024.
Björn Engelmann, Timo Breuer, Jana Isabelle Friese, Philipp Schaer and Norbert Fuhr.
[pdf]  [BibTeX] 

2023

Bibliometric Data Fusion for Biomedical Information Retrieval.
In: ACM/IEEE Joint Conference on Digital Libraries, JCDL 2023, Santa Fe, NM, USA, June 26-30, 2023, pages 107-118. IEEE, 2023.
Timo Breuer, Christin Katharina Kreutz, Philipp Schaer and Dirk Tunger.
[doi] [pdf]  [BibTeX] 
An in-depth investigation on the behavior of measures to quantify reproducibility.
Information Processing and Management, 60(3):103332, 2023.
Maria Maistro, Timo Breuer, Philipp Schaer and Nicola Ferro.
[doi] [pdf]  [BibTeX]