Entity tracking in language models Keeping track of how states and relations of entities change as a text or dialog unfolds is a key prerequisite to discourse understanding as well as other AI tasks such as planning, and yet it remains unclear to what extent pretrained language models systematically exhibit this capability. In my talk, I will first discuss the challenges that come with evaluating such general abilities in LMs, and then I will present a new evaluation task for assessing entity tracking abilities in LMs. I will then present results on GPT-3/3.5/4, Flan-T5, and Llama 2 models and discuss the influence of pretraining on code for entity tracking abilities. I will also show that smaller models can learn to track entities but their generalization abilities are still quite limited, and present some preliminary results from probing experiments that provide some insights into how different models solve this task.
Sebastian Schuster is a lecturer in computational linguistics at University College London. His main research interest is in computational semantics and pragmatics, particularly developing formal and psycholinguistic accounts of pragmatic language use and more reliable natural language understanding systems.