Publications
Browsing and Searching Metadata of TREC.
In:
Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, series SIGIR '24, pages 313–323.
Association for Computing Machinery, New York, NY, USA, 2024.
Timo Breuer, Ellen M. Voorhees and Ian Soboroff.
[doi] [pdf]
[abstract]
[BibTeX]
Information Retrieval (IR) research is deeply rooted in experimentation and evaluation, and the Text REtrieval Conference (TREC) has been playing a central role in making that possible since its inauguration in 1992. TREC's mission centers around providing the infrastructure and resources to make IR evaluations possible at scale. Over the years, a plethora of different retrieval problems were addressed, culminating in data artifacts that remained as valuable and useful tools for the IR community. Even though the data are largely available from TREC's website, there is currently no resource that facilitates a cohesive way to obtain metadata information about the run file - the IR community's de-facto standard data format for storing rankings of system-oriented IR experiments.To this end, the work at hand introduces a software suite that facilitates access to metadata of experimental resources, resulting from over 30 years of IR experiments and evaluations at TREC. With a particular focus on the run files, the paper motivates the requirements for better access to TREC metadata and details the concepts, the resources, the corresponding implementations, and possible use cases. More specifically, we contribute a web interface to browse former TREC submissions. Besides, we provide the underlying metadatabase and a corresponding RESTful interface for more principled and structured queries about the TREC metadata.
Toward Evaluating the Reproducibility of Information Retrieval Systems with Simulated Users.
In:
Proceedings of the 2nd ACM Conference on Reproducibility and Replicability, series ACM REP '24, pages 25–29.
Association for Computing Machinery, New York, NY, USA, 2024.
Timo Breuer and Maria Maistro.
[doi] [pdf]
[abstract]
[BibTeX]
Reproducibility is a fundamental part of scientific progress. Compared to other scientific fields, computational sciences are privileged as experimental setups can be preserved with ease, and regression experiments allow the validation of computational results by bitwise similarity. When evaluating information access systems, the system users are often considered in the experiments, be it explicit as part of user studies or implicit as part of evaluation measures. Usually, system-oriented Information Retrieval (IR) experiments are evaluated with effectiveness measurements and batches of multiple queries. Successful reproduction of an Information Retrieval (IR) system is often determined by how well it approximates the averaged effectiveness of the original (reproduced) system. Earlier work suggests that the nave comparison of average effectiveness hides differences that exist between the original and reproduced systems. Most importantly, such differences can affect the recipients of the retrieval results, i.e., the system users. To this end, this work sheds light on what implications for users may be neglected when a system-oriented Information Retrieval (IR) experiment is prematurely considered reproduced. Based on simulated reimplementations with comparable effectiveness as the reference system, we show what differences are hidden behind averaged effectiveness scores. We discuss possible future directions and consider how these implications could be addressed with user simulations.
Validating Synthetic Usage Data in Living Lab Environments.
Journal of Data and Information Quality, 16(1):1-33, 2024.
Timo Breuer, Norbert Fuhr and Philipp Schaer.
[doi] [pdf]
[BibTeX]
Context-Driven Interactive Query Simulations Based on Generative Large Language Models.
In:
ECIR 2024.
2024.
Björn Engelmann, Timo Breuer, Jana Isabelle Friese, Philipp Schaer and Norbert Fuhr.
[pdf]
[BibTeX]
Replicability Measures for Longitudinal Information Retrieval Evaluation.
In:
Experimental IR Meets Multilinguality, Multimodality, and Interaction - 15th International Conference of the CLEF Association, CLEF 2024, Grenoble, France, September 9–12, 2024, Proceedings, Part I.
Springer Cham, 2024.
Jüri Keller, Timo Breuer and Philipp Schaer.
[pdf]
[BibTeX]
SIGIR 2024 Workshop on Simulations for Information Access (Sim4IA 2024).
In:
Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, series SIGIR '24, pages 3058–3061.
Association for Computing Machinery, New York, NY, USA, 2024.
Philipp Schaer, Christin Katharina Kreutz, Krisztian Balog, Timo Breuer and Norbert Fuhr.
[doi] [pdf]
[abstract]
[BibTeX]
Simulations in various forms have been used to evaluate information access systems, like search engines, recommender systems, or conversational agents. In the form of the Cranfield paradigm, a simulation setup is well-known in the IR community, but user simulations have recently gained interest. While user simulations help to reduce the complexity of evaluation experiments and help with reproducibility, they can also contribute to a better understanding of users. Building on recent developments in methods and toolkits, the Sim4IA workshop aims to bring together researchers and practitioners to form an interactive and engaging forum for discussions on the future perspectives of the field. An additional aim is to plan an upcoming TREC/CLEF campaign.
Bibliometric Data Fusion for Biomedical Information Retrieval.
In:
ACM/IEEE Joint Conference on Digital Libraries, JCDL 2023, Santa Fe, NM, USA, June 26-30, 2023, pages 107-118.
IEEE, 2023.
Timo Breuer, Christin Katharina Kreutz, Philipp Schaer and Dirk Tunger.
[doi] [pdf]
[BibTeX]
An in-depth investigation on the behavior of measures to quantify reproducibility.
Information Processing and Management, 60(3):103332, 2023.
Maria Maistro, Timo Breuer, Philipp Schaer and Nicola Ferro.
[doi] [pdf]
[BibTeX]