STELLA II - Infrastructures for Living Labs

Profile & Description

The STELLA project provides an innovative technology and methodology infrastructure that allows information providers to evaluate their information systems with the actual users of their web platforms. The underlying principle of the Living Labs allows STELLA to interpose itself between the communication of the users with the systems and thus not to evaluate in an artificial and static laboratory environment (as usual in so-called TREC studies), but to pick up the users there, where they satisfy their actual information needs. By incorporating the Living Lab principle, the systems can be evaluated iteratively and continuously, thus taking a big step towards a “Continues Evaluation”, as is already common practice with large commercial web platforms (such as Amazon). After a successful test run of the platform at the two specialist information providers GESIS and ZB MED and the implementation of an evaluation campaign in the context of the international conference CLEF, a special focus will be placed on the continuation and dissemination of the evaluation infrastructure in project phase II. To this end, the project will integrate further external providers from the German-language specialist information landscape, into whose systems STELLA will be implemented. In order to make it as easy as possible for the partners to get started, the STELLA II team will take on the role of a technical and methodological consultant and will also carry out consulting and development work on site. In order to anchor the Living Labs not only technically but also methodologically further into the professional information landscape, special attention will be paid to the principle of reproducibility and comparability of the experiments.

ProjectSTELLA II - Infrastructures for Living Labs

Duration
2023 - 2025
Funded by

External Resources

Publications

2023

An in-depth investigation on the behavior of measures to quantify reproducibility.
Information Processing and Management, 60(3):103332, 2023.
Maria Maistro, Timo Breuer, Philipp Schaer and Nicola Ferro.
[doi] [pdf]  [BibTeX] 
Preliminary Results of a Scientometric Analysis of the German Information Retrieval Community 2020-2023.
In: M. Leyer and J. Wichmann, editors, Proceedings of the LWDA 2023 Workshops: BIA, DB, IR, KDML and WM. Marburg, Germany, 09.-11. October 2023, volume 3630, series CEUR Workshop Proceedings, pages 222-230. CEUR-WS.org, 2023.
Philipp Schaer, Svetlana Myshkina and Jüri Keller.
[doi] [pdf]  [BibTeX]