Large Language Models, Spring 2023

Course Description

Large language models have become one of the most commonly deployed NLP inventions. In the past half-decade, their integration into core natural language processing tools has dramatically increased the performance of such tools, and they have entered the public discourse surrounding artificial intelligence. In this course, we offer a self-contained introduction to language modeling and its applications. We start with the probabilistic foundations of language models, i.e., covering what constitutes a language model from a formal, theoretical perspective. We then discuss how to construct and curate training corpora, and introduce many of the neural-network architectures often used to instantiate language models at scale. The course covers aspects of systems programming, discussion of privacy and harms, as well as applications of language models in NLP and beyond.

News

3. 1. 2023 Class website is online!

Syllabus and Schedule

On the Use of Class Time

Lectures

There are two lecture slots for LLM each week: the first one on Tuesdays 14-16 in HG E 1.2 and the second one on Fridays 10-11 in CAB G 61.

Both lectures will be given in person and live broadcast on Zoom; the password is available on the course Moodle page.

Lectures will be recorded—links to the Zoom recordings will be posted on the course Moodle page.

Discussion Sections

Discussion sections (tutorials) will take place Thursdays 16-18 in NO C 60 and on Zoom (same link as the lectures).

Syllabus

Date	Module	Topic	Lecturer	Summary	Reading
21. 2. 2023		Introduction and Overview	Ryan/Mrinmaya/Ce/Florian	The lecturers will contextualize large language models in NLP and computer science more broadly. Thereby, we will also motivate why the topic necessitates a separate course. We will also go over the course schedule and logistics.
21. 2. 2023	Probabilistic Foundations	Basic Measure Theory	Ryan	Language modeling is about placing probability on infinite sets of strings. Measure theory is the primary tool used for the rigorous study of probability theory. This lecture shows why defining a language model rigorously requires a careful measure-theoretic treatment. We use the classic infinite coin toss model as an illuminating example. Then, we will get into some basic measure-theoretic definitions that will be useful in formally defining language models.	Du, Li, et al. A Measure-Theoretic Characterization of Tight Language Models. arXiv, 2022.
24. 2. 2023		Defining a Language Model	Ryan	We will continue to introduce definitions and facts from basic measure theory, building up to a formal definition of a language model, which will be our working definition throughout the class.
28. 2. 2023		Tight Language Models	Ryan	The primary goal of this lecture is to introduce the notion of tightness, which will be a recurring theoretical concept in the first part of the course. Informally, a language model is tight when it only places probability mass on finite strings. We introduce the Borel-Cantelli lemmata and prove a precise characterization of tight language models.	Du, Li, et al. A Measure-Theoretic Characterization of Tight Language Models. arXiv, 2022., Chen, Yining, et al. Recurrent Neural Networks as Weighted Language Recognizers. arXiv, 2017.
3. 3. 2023	Modeling Foundations	The Language Modeling Task	Ryan	In this lecture, we introduce the language modeling task, which we define to be any attempt to learn a language model from finite data. We will discuss various objectives that one might wish to optimize to induce a language model from data. We also discuss various regularization techniques and their use in combatting overfitting.
7. 3. 2023		Finite-State Language Models	Ryan	Finite-state language models have a storied history in NLP. They are a natural generalization of n-gram models, which were the standard in the field from the 1980s till the late 2010s. In terms of theory, we introduce probabilistic finite-state automata as a generalization of finite-state automata from classic theory of computation. Additionally, we give a simple, closed-form characterization of tightness. We also show how Bengio et al. (2003), the first successful neural language model, is naturally viewed as a probabilistic finite-state automaton.	Bengio, Yoshua, et al. A neural probabilistic language model. J. Mach. Learn. Res., 2003.
10. 3. 2023		Pushdown Language Models	Ryan	In many ways, human language is more naturally modeled by a context-free grammar than by a finite-state automaton. This lecture discusses how to use weighted context-free grammars, specifically when implemented as weighted pushdown automata, to construct language models. In the case of a 1-stack pushdown language model, we give an iterative algorithm to determine tightness. We also discuss pushdown language models with more than one stack. In this case, determining whether such a language model is tight is undecidable. Learning the nuts and bolts of pushdown language models is more than just a historical artifact: The definitions provided in this lecture will serve as a basis for proofs about the capacity of recurrent neural networks. Indeed, our proof that it is undecidable to determine the tightness of a recurrent neural language model with infinite precision is as simple as demonstrating an encoding of a 2-stack pushdown language model as a recurrent neural network.
14. 3. 2023	Neural Network Modeling	Recurrent Neural Language Models	Ryan	Finite-state language models, by construction, can only look at a finite amount of context. Recurrent neural networks are a formalism that overcomes this limitation. In this lecture, we give a formal definition of a recurrent neural language model (RNNLM). We give examples of tight and non-tight RNNLMs as well as characterize the vanishing gradient problem.
17. 3. 2023		Variants of RNNLMs	Ryan	We discuss several popular variants of the RNN, most notably the LSTM and GRU. We give a formal argument showing that these variants mitigate the vanishing gradient problem.
21. 3. 2023		Representational Capacity of RNNLMs	Ryan	In this lecture, we explore the representational capacity of RNNLMs. We show that, if the activation function is a hard thresholding operation, then RNNLMs have the same expressive capacity as a finite-state LM. However, we show that RNNLMs can implicitly represent finite-state LMs that are much larger. Additionally, if the activation function is a saturated sigmoid or a ReLu and we assume infinite precision arithmetic, we show how an RNN can emulate a Turing machine.	Siegelmann H. T. and Sontag E. D. On the computational power of neural nets. Computational learning theory. 1992.
24. 3. 2023		Transformer-based Language Models	Ryan	Introduced in 2017 by Vaswani et al., Transformers have quickly become the most popular architecture for neural language modeling. They are the basis for recent large language models, e.g., GPT-3 and PaLM. This lecture gives the definition of a Transformer and overviews details, e.g., residual connections, layer normalization, and position embeddings.
28. 3. 2023		Efficient Attention	Ryan	There is an ever-growing bag of tricks that speed up the computation of the attention mechanism in Transformer-based language models. This lecture overview those tricks, which are becoming increasingly necessary to scale up Transformer LMs on academic hardware. Where possible, we prove guarantees for the methods.
31. 3. 2023		Variants on the Transformer	Ryan	This lecture overviews various generalizations of the transformer. We will discuss multi-headed attention, sparse attention, and Transformer variants tailored for long documents.
4. 4. 2023	Modeling Potpourri	Tokenization	Ryan	Throughout the class, we have assumed access to the alphabet Σ. This lecture discusses how we should choose Σ. We discuss various facts about natural language that influence Σ, e.g., morphology and syntax. Then, we introduce the byte-pair encoding algorithm, an automatic procedure for inducing Σ, and give a analyze of its correctness and runtime.
	Easter Break
18. 4. 2023	Modeling Potpourri	Generating Text from a Language Model	Ryan	A popular use case for language modeling is the generation of text. This lecture overviews various strategies for deterministically and stochastically generating text. We discuss beam search, ancestral sampling, as well as various sampling adaptors, e.g., top-k, nucleus, and locally typical sampling.
21. 4. 2023	Parallelism and Scaling up	Scaling up	Ce
25. 4. 2023	Parallelism and Scaling up	Parallelism	Ce
28. 4. 2023	Applications	Pre-training	Mrinmaya
2. 5. 2023		Fine-Tuning	Mrinmaya
5. 5. 2023		Multimodality	Mrinmaya
9. 5. 2023		Prompting	Mrinmaya
12. 5. 2023		Additional Topics	Mrinmaya
16. 5. 2023	Analysis	Analysis and Probing	Tiago/Ryan	Many language models are uninterpretable, i.e., it is hard to know why a language model prefers one prediction to another. This lecture overviews a variety of recent techniques for better understanding language models’ behavior and interpreting their predictions.
19. 5. 2023	Analysis	Cognitive Modeling	Ethan/Alex/Ryan	Language models show remarkable linguistic capabilities. This lecture treats the question: Do language models process language as humans do? The performance of language modeling on a wide variety of cognitive benchmarks is discussed in an attempt to tease apart how language models are similar and dissimilar to human language processing. We will also discuss the implication of language remodeling on language science.
23. 5. 2023	Security and Misuse	Harms	Florian
26. 5. 2023		Ethical Concerns	Florian
30. 5. 2023		Security and Misuse	Florian
2. 6. 2023		Memorization and Privacy	Florian

Organisation

Live Chat

In addition to class time, there will also be a RocketChat-based live chat hosted on ETH’s servers. Students are free to ask questions of the teaching staff and of others in public or private (direct message). There are specific channels for each of the two assignments as well as for reporting errata in the course notes and slides. All data from the chat will be deleted from ETH servers at the course’s conclusion.

Important: There are a few important points you should keep in mind about the course live chat:

RocketChat will be the main communications hub for the course. You are responsible for receiving all messages broadcast in the RocketChat.
Your username should be firstname.lastname. This is required as we will only allow enrolled students to participate in the chat and we will remove users which we cannot validate.
Tag your questions as described in the document on How to use Rycolab Course RocketChat channels. The document also contains other general remarks about the use of RocketChat.
Search for answers in the appropriate channels before posting a new question.
Ask questions on public channels as much as possible.
Answer to posts in threads.
The chat supports LaTeX for easier discussion of technical material. See How to use LaTeX in RocketChat.
We highly recommend you download the desktop app here.

This is the link to the main channel. To make the moderation of the chat more easily manageable, we have created a number of other channels on RocketChat. The full list is:

LLM General Channel for the general organisational discussions.
LLM Announcements Channel for the announcements by the teaching team.
LLM Content Questions for your questions about the content of the course.
LLM Errata for reporting typos and errors in the course lecture notes and the slides.
LLM Assignment 1 for discussing and asking questions about Assignment 1.
LLM Assignment 2 for discussing and asking questions about Assignment 2.
Find Assignment Partners for finding teammates for the course assignments.

If you feel like you would benefit from any other channel, feel free to suggest it to the teaching team!

Course Notes

We will prepare the course lecture notes as we go! The individual chapters will be published in the course syllabus and updated throughout the semester. Please report all errata to the teaching staff; we created an errata channel in RocketChat.

Other useful literature:

Grading

Marks for the course will be determined by the following formula:

50% Final Exam
50% Assignments

On the Final Exam

The final exam is comprehensive and should be assumed to cover all the material in the slides and class notes.

On the Class Assignments

There will be 2 larger assignments in the course.

We require the solutions to be properly typeset. We recommend using LaTeX (with Overleaf), but markdown files with MathJax for the mathematical expressions are also fine.

Large Language Models, Spring 2023

Course Description

News

Syllabus and Schedule

On the Use of Class Time

Lectures

Discussion Sections

Syllabus

Organisation

Live Chat

Course Notes

Grading

On the Final Exam

On the Class Assignments

Large Language Models Lecturers

Assistant Professor in Computer Science

Assistant Professor in Computer Science

Assistant Professor in Computer Science

Assistant Professor of Computer Science

Large Language Models Guest Lecturers

Postdoc

Postdoc

PhD Student

Large Language Models Teaching Assistants

(incoming) PhD Student

PhD Student

PhD Student

Master’s Student

PhD Student/Web Meister

PhD Student

PhD Student