STELLA II - Infrastructures for Living Labs

Profile & Description

The STELLA project provides an innovative technology and methodology infrastructure that allows information providers to evaluate their information systems with the actual users of their web platforms. The underlying principle of the Living Labs allows STELLA to interpose itself between the communication of the users with the systems and thus not to evaluate in an artificial and static laboratory environment (as usual in so-called TREC studies), but to pick up the users there, where they satisfy their actual information needs. By incorporating the Living Lab principle, the systems can be evaluated iteratively and continuously, thus taking a big step towards a “Continues Evaluation”, as is already common practice with large commercial web platforms (such as Amazon). After a successful test run of the platform at the two specialist information providers GESIS and ZB MED and the implementation of an evaluation campaign in the context of the international conference CLEF, a special focus will be placed on the continuation and dissemination of the evaluation infrastructure in project phase II. To this end, the project will integrate further external providers from the German-language specialist information landscape, into whose systems STELLA will be implemented. In order to make it as easy as possible for the partners to get started, the STELLA II team will take on the role of a technical and methodological consultant and will also carry out consulting and development work on site. In order to anchor the Living Labs not only technically but also methodologically further into the professional information landscape, special attention will be paid to the principle of reproducibility and comparability of the experiments.

ProjectSTELLA II - Infrastructures for Living Labs

Duration
2023 - 2025
Funded by

External Resources

Publications

2024

Evaluation of Temporal Change in IR Test Collections.
In: Proceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval, series ICTIR '24, pages 3–13. Association for Computing Machinery, New York, NY, USA, 2024.
Jüri Keller, Timo Breuer and Philipp Schaer.
[doi] [pdf]  [abstract]  [BibTeX] 
Leveraging Prior Relevance Signals in Web Search.
In: G. Faggioli, N. Ferro, P. Galuscáková and A. G. S. de Herrera, editors, Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024), Grenoble, France, 9-12 September, 2024, volume 3740, series CEUR Workshop Proceedings, pages 2396-2406. CEUR-WS.org, 2024.
Jüri Keller, Timo Breuer and Philipp Schaer.
[doi]  [BibTeX] 
Replicability Measures for Longitudinal Information Retrieval Evaluation.
In: Experimental IR Meets Multilinguality, Multimodality, and Interaction - 15th International Conference of the CLEF Association, CLEF 2024, Grenoble, France, September 9–12, 2024, Proceedings, Part I. Springer Cham, 2024.
Jüri Keller, Timo Breuer and Philipp Schaer.
[pdf]  [BibTeX] 
SIGIR 2024 Workshop on Simulations for Information Access (Sim4IA 2024).
In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, series SIGIR '24, pages 3058–3061. Association for Computing Machinery, New York, NY, USA, 2024.
Philipp Schaer, Christin Katharina Kreutz, Krisztian Balog, Timo Breuer and Norbert Fuhr.
[doi] [pdf]  [abstract]  [BibTeX] 

2023

Evaluating Temporal Persistence Using Replicability Measures.
In: M. Aliannejadi, G. Faggioli, N. Ferro and M. Vlachos, editors, Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2023), Thessaloniki, Greece, September 18th to 21st, 2023, volume 3497, series CEUR Workshop Proceedings, pages 2441-2457. CEUR-WS.org, 2023.
Jüri Keller, Timo Breuer and Philipp Schaer.
[doi]  [BibTeX] 
Preliminary Results of a Scientometric Analysis of the German Information Retrieval Community 2020-2023.
In: M. Leyer and J. Wichmann, editors, Proceedings of the LWDA 2023 Workshops: BIA, DB, IR, KDML and WM. Marburg, Germany, 09.-11. October 2023, volume 3630, series CEUR Workshop Proceedings, pages 222-230. CEUR-WS.org, 2023.
Philipp Schaer, Svetlana Myshkina and Jüri Keller.
[doi] [pdf]  [BibTeX]