RESIRE - Reproducibility and simulation of interactive retrieval experiments

Profile & Description

Although reproducibility of experiments is a topic with increasing importance in information retrieval (IR), there has been hardly any work on reproducing interactive IR experiments. The main idea of our proposed project is to describe actual users by quantitative modeling of their behavior and their content-related decisions during retrieval sessions. These model parameters can then form the basis for comparing two interactive IR experiments with regard to reproducibility, and can also be used for simulating interactive IR, e. g. for evaluating new systems or system variants without (or before) carrying out actual user experiments. As part of the proposed project, we will define a new framework model for reproducibility of interactive IR experiments, develop new metrics for reproducibility and session quality, design new simulation methods integrating behavior and content-related decisions, develop an open source framework for reproduction and simulation of interactive IR, and perform extensive experiments using existing IR log data as well as observation data from new user experiments.

Funding Agency
DFG - Deutsche Forschungsgemeinschaft
Partner Institution
University of Duisburg-Essen
People Involved
Prof. Dr. Philipp Schaer (Technische Hochschule Köln)
Andreas Kruff (Technische Hochschule Köln)

ProjectRESIRE - Reproducibility and simulation of interactive retrieval experiments

Duration
2023 - 2026
Funded by

Publications

2024

Browsing and Searching Metadata of TREC.
In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, series SIGIR '24, pages 313–323. Association for Computing Machinery, New York, NY, USA, 2024.
Timo Breuer, Ellen M. Voorhees and Ian Soboroff.
[doi] [pdf]  [abstract]  [BibTeX] 
Toward Evaluating the Reproducibility of Information Retrieval Systems with Simulated Users.
In: Proceedings of the 2nd ACM Conference on Reproducibility and Replicability, series ACM REP '24, pages 25–29. Association for Computing Machinery, New York, NY, USA, 2024.
Timo Breuer and Maria Maistro.
[doi] [pdf]  [abstract]  [BibTeX] 
Validating Synthetic Usage Data in Living Lab Environments.
Journal of Data and Information Quality, 16(1):1-33, 2024.
Timo Breuer, Norbert Fuhr and Philipp Schaer.
[doi] [pdf]  [BibTeX] 
Context-Driven Interactive Query Simulations Based on Generative Large Language Models.
In: ECIR 2024. 2024.
Björn Engelmann, Timo Breuer, Jana Isabelle Friese, Philipp Schaer and Norbert Fuhr.
[pdf]  [BibTeX] 
Replicability Measures for Longitudinal Information Retrieval Evaluation.
In: Experimental IR Meets Multilinguality, Multimodality, and Interaction - 15th International Conference of the CLEF Association, CLEF 2024, Grenoble, France, September 9–12, 2024, Proceedings, Part I. Springer Cham, 2024.
Jüri Keller, Timo Breuer and Philipp Schaer.
[pdf]  [BibTeX] 
SIGIR 2024 Workshop on Simulations for Information Access (Sim4IA 2024).
In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, series SIGIR '24, pages 3058–3061. Association for Computing Machinery, New York, NY, USA, 2024.
Philipp Schaer, Christin Katharina Kreutz, Krisztian Balog, Timo Breuer and Norbert Fuhr.
[doi] [pdf]  [abstract]  [BibTeX] 

2023

Bibliometric Data Fusion for Biomedical Information Retrieval.
In: ACM/IEEE Joint Conference on Digital Libraries, JCDL 2023, Santa Fe, NM, USA, June 26-30, 2023, pages 107-118. IEEE, 2023.
Timo Breuer, Christin Katharina Kreutz, Philipp Schaer and Dirk Tunger.
[doi] [pdf]  [BibTeX] 
An in-depth investigation on the behavior of measures to quantify reproducibility.
Information Processing and Management, 60(3):103332, 2023.
Maria Maistro, Timo Breuer, Philipp Schaer and Nicola Ferro.
[doi] [pdf]  [BibTeX]