Chat about SOEP Research Data: A RAG System for Interactive SOEP Data Exploration

Publication details

ABSTRACT
The German Socio-Economic Panel (SOEP) is one of the world’s longest-running household
panel studies, containing rich longitudinal data spanning over four decades. However, the
complexity and scale of SOEP data present significant challenges for researchers in data
discovery and exploration. This paper presents a prototype Retrieval-Augmented Generation
(RAG) system designed to provide conversational access to SOEP research data through natural language queries. Built on OpenWebUI with PostgreSQL and Qdrant vector databases, while leveraging open-weight LLMs, our system enables researchers to interactively explore dataset descriptions, variable definitions, survey questions, and metadata. While currently covering a substantial portion of SOEP datasets, this prototype demonstrates the potential for conversational AI to enhance research data accessibility and streamline the data discovery process for panel data researchers.

Keywords: LLM, AI, RAG, variable search, metadata, DDI