The Friedrich Schiller University Jena (FUSION group) seeks to fill the thesis position of a:
Bachelor/ Master thesis (m/f/d)
Evaluation of Multi-Agent Retrieval-Augmented Generation (RAG) Systems Without Ground Truth
* commencing at the earliest opportunity
* Place of work: Jena, FUSION group at the Friedrich Schiller University Jena, Work from home is possible
Background:
Friedrich Schiller University is a traditional university with a strong research profile rooted in the heart of Germany. As a university covering all disciplines, it offers a wide range of subjects. Its research is focused on the areas Light—Life—Liberty. It is closely networked with non-research institutions, research companies and renowned cultural institutions. With around 18,000 students and more than 8,600 employees, the university plays a major role in shaping Jena's character as a cosmopolitan and future-oriented city.
The project:
The position is affiliated with the FUSION group of Univ. Prof. Dr Birgitta König-Ries, at Friedrich Schiller University Jena and in collaboration with the Nixi AI ). The FUSION group is highly interdisciplinary and diverse, working towards building better biodiversity platforms and tools. More information about FUSION: Nixi AI is a health tech startup which is an all-in-one platform for AI-powered medical documentation, billing, and decision support.
The project focuses on building and validating a multi-stage RAG (Retrieval‑Augmented Generation) pipeline without relying on any pre-existing ground truth data. It will explore two complementary evaluation methods: generating synthetic question-answer pairs with LLMs to serve as pseudo-ground truth, and employing an "LLM-as-a-judge" approach, where one or more LLMs assess the relevance, factual accuracy, and fluency of the pipeline's outputs based on carefully designed prompts. A primary objective is to produce a confidence score for each generated answer, enabling end users to trust the system's responses. The evaluation framework will be implemented in Python, integrating judge agents and synthetic benchmarks, and will be calibrated and validated through human spot-checks and statistical analyses to ensure the calibration of confidence and reliability of judgments.
Your tasks:
* Build a multi-agent RAG in Python, orchestrating LLM agent calls.
* Design evaluation rubrics and structured outputs.
* Conduct prompt tuning and calibration experiments (along with creating synthetic GT data).
* Optionally, wrap components in a chatbot interface for live test interactions.
Your profile:
* Student in the fields of computer science, mathematics or any comparable degree
* Experience with the Python programming language
* Prior knowledge of working with LLMs, Langchain, Langgraph, and Huggingface is a plus
* Capable of working independently
* Willingness to learn new skills and technologies
* Fluency in English and good communication skills
We offer:
* Work in a dynamic, international, and interdisciplinary environment in the beautiful city of Jena
* Flexible working hours and a family-friendly working environment
* Please note: As this is an academic thesis in direct collaboration with the university, the position is unpaid.
Since all application documents will be duly destroyed after the recruitment process, we ask you to submit only copies of your documents.
Interested? Send us your CV and cover letter at until 30 August, 2025.
In your cover letter, please confirm the following:
* That your university permits an externally supervised master's thesis.
* That you understand this academic position is unpaid.
This will help streamline the application process.
For further information for applicants, please also refer to (in German)and Please also note the information on the collection of personal data