Evaluating the interactions between users and systems presents many challenges. Simulation offers a reliable, re-usable, and repeatable methodology to explore how different users, user behaviours and/or retrieval systems impact performance. With Large Language Models and Generative AI now widely available and accessible, new affordances are possible. These allow researchers to create more ''realistic'' simulated users that can generate queries and judge items like humans, and to develop new retrieval systems where responses and interactions are conversational and based on retrieval augmented generation. This resource paper presents a community-led initiative to update the Simulation of Interactive Information Retrieval (SimIIR) Framework to enable the simulation of conversational search using LLMs. The largest update provides a conversational search workflow which involves a number of new possible interactions with a search system or agent -- enabling a host of new development and evaluation opportunities. Other developments include the Markovian Users, Cognitive States, LLM-based components for assessing snippets/documents/responses, generating queries, deciding on when to stop/continue, and PyTerrier integration. This paper aims to mark the release of SimIIR 3.0 and invites the community to build, extend, and use the resource.

Evaluation of text simplification currently focuses on the difference of a source text to its simplified variant. Datasets for this evaluation base on a specific topic and group of readers for which is simplified. The broad applicability of text simplification and specifics that come with intended target audiences (e.g., children compared to adult non-experts) are disregarded. An explainable assessment of the overall simplicity of text is missing. This work is BenchmArking Text Simplicity (BATS): we provide an explainable method to assess practical and concrete rules from literature describing features of simplicity and complexity of text. Our experiments on 15 datasets for text simplification highlight differences in features that are important in different domains of text and for different intended target audiences.

Simulations in various forms have been used to evaluate information access systems, like search engines, recommender systems, or conversational agents. In the form of the Cranfield paradigm, a simulation setup is well-known in the IR community, but user simulations have recently gained interest. While user simulations help to reduce the complexity of evaluation experiments and help with reproducibility, they can also contribute to a better understanding of users. Building on recent developments in methods and toolkits, the Sim4IA workshop aims to bring together researchers and practitioners to form an interactive and engaging forum for discussions on the future perspectives of the field. An additional aim is to plan an upcoming TREC/CLEF campaign.