Generating Text from Language Models

ACL 2023

RocketChat Channel

Tutorial Description

An increasingly large percentage of natural language processing (NLP) tasks center around the generation of text from probabilistic language models. Despite this trend, techniques for improving or specifying preferences in these generated texts rely mostly on intuition-based heuristics. Further, there lacks a unified presentation of their motivations, practical implementation, successes and pitfalls. Practitioners must, therefore, choose somewhat blindly between generation algorithms—like top-p sampling or beam search—which can lead to wildly different results. At the same time, language generation research continues to criticize and improve the standard toolboxes, further adding entropy to the state of the field. In this tutorial, we will provide a centralized and cohesive discussion of critical considerations when choosing how to generate from a language model. We will cover a wide range of empirically-observed problems (like degradation, hallucination, repetition) and their corresponding proposed algorithmic solutions from recent research (like top-p sampling and its successors). We will then discuss a subset of these algorithms under a unified light; most stochastic generation strategies can be framed as locally adapting the probabilities of a model to avoid failure cases. Finally, we will then cover methods in controlled generation, that go beyond just ensuring coherence to ensure text exhibits specific desired properties. We aim for NLP practitioners and researchers to leave our tutorial with a unified framework which they can use to evaluate and contribute to the latest research in language generation.


Slide deck will be continually updated.


Module 1 Probability Distributions Over Strings
Module 2 Successes and Failures of Estimating Language Models
Module 3 Basics of Generating Text Colab Notebook
Module 4 Sampling Methods Colab Notebook
Module 5 Controlled Generation Colab Notebook
Module 6 Evaluating Language Generators Colab Notebook
Module 7 Looking Forward...

Suggested Literature

Probability distributions over strings

Successes and Failures of Estimating Language Models

Sampling Methods

Controlled Generation

Measuring the quality of language generators and its challenges