Large Language Models, Spring 2023
ETH Zürich: Course catalog
Large language models have become one of the most commonly deployed NLP inventions. In the past half-decade, their integration into core natural language processing tools has dramatically increased the performance of such tools, and they have entered the public discourse surrounding artificial intelligence. In this course, we offer a self-contained introduction to language modeling and its applications. We start with the probabilistic foundations of language models, i.e., covering what constitutes a language model from a formal, theoretical perspective. We then discuss how to construct and curate training corpora, and introduce many of the neural-network architectures often used to instantiate language models at scale. The course covers aspects of systems programming, discussion of privacy and harms, as well as applications of language models in NLP and beyond.
3. 1. 2023 Class website is online!
Syllabus and Schedule
On the Use of Class Time
Lectures will be recorded—links to the Zoom recordings will be posted on the course Moodle page.
|21. 2. 2023||Introduction and Overview||Ryan/Mrinmaya/Ce/Florian|
|21. 2. 2023||Probabilistic Foundations||Basic Measure Theory||Ryan||Du, Li, et al. A Measure-Theoretic Characterization of Tight Language Models. arXiv, 2022.|
|24. 2. 2023||Defining a Language Model||Ryan|
|28. 2. 2023||Tight Language Models||Ryan||Du, Li, et al. A Measure-Theoretic Characterization of Tight Language Models. arXiv, 2022., Chen, Yining, et al. Recurrent Neural Networks as Weighted Language Recognizers. arXiv, 2017.|
|3. 3. 2023||Modeling Foundations||The Language Modeling Task||Ryan|
|7. 3. 2023||Finite-State Language Models||Ryan||Bengio, Yoshua, et al. A neural probabilistic language model. J. Mach. Learn. Res., 2003.|
|10. 3. 2023||Pushdown Language Models||Ryan|
|14. 3. 2023||Neural Network Modeling||Recurrent Neural Language Models||Ryan|
|17. 3. 2023||Variants of RNNLMs||Ryan|
|21. 3. 2023||Representational Capacity of RNNLMs||Ryan||Siegelmann H. T. and Sontag E. D. On the computational power of neural nets. Computational learning theory. 1992.|
|24. 3. 2023||Transformer-based Language Models||Ryan|
|28. 3. 2023||Efficient Attention||Ryan|
|31. 3. 2023||Variants on the Transformer||Ryan|
|4. 4. 2023||Modeling Potpourri||Tokenization||Ryan|
|18. 4. 2023||Modeling Potpourri||Generating Text from a Language Model||Ryan|
|21. 4. 2023||Parallelism and Scaling up||Scaling up||Ce|
|25. 4. 2023||Parallelism||Ce|
|28. 4. 2023||Applications||Pre-training||Mrinmaya|
|2. 5. 2023||Fine-Tuning||Mrinmaya|
|5. 5. 2023||Multimodality||Mrinmaya|
|9. 5. 2023||Prompting||Mrinmaya|
|12. 5. 2023||Additional Topics||Mrinmaya|
|16. 5. 2023||Analysis||Analysis and Probing||Tiago/Ryan|
|19. 5. 2023||Cognitive Modeling||Ethan/Alex/Ryan|
|23. 5. 2023||Security and Misuse||Harms||Florian|
|26. 5. 2023||Ethical Concerns||Florian|
|30. 5. 2023||Security and Misuse||Florian|
|2. 6. 2023||Memorization and Privacy||Florian|
In addition to class time, there will also be a
RocketChat-based live chat hosted on ETH’s servers.
Students are free to ask questions of the teaching staff and of others in public or private (direct message).
There are specific channels for each of the two assignments as well as for reporting errata in the course notes and slides.
All data from the chat will be deleted from ETH servers at the course’s conclusion.
Important: There are a few important points you should keep in mind about the course live chat:
RocketChatwill be the main communications hub for the course. You are responsible for receiving all messages broadcast in the
- Your username should be
firstname.lastname. This is required as we will only allow enrolled students to participate in the chat and we will remove users which we cannot validate.
- Tag your questions as described in the document on How to use Rycolab Course RocketChat channels. The document also contains other general remarks about the use of
- Search for answers in the appropriate channels before posting a new question.
- Ask questions on public channels as much as possible.
- Answer to posts in threads.
- The chat supports
LaTeXfor easier discussion of technical material. See How to use
- We highly recommend you download the desktop app here.
This is the link to the main channel.
To make the moderation of the chat more easily manageable, we have created a number of other channels on
The full list is:
- LLM General Channel for the general organisational discussions.
- LLM Announcements Channel for the announcements by the teaching team.
- LLM Content Questions for your questions about the content of the course.
- LLM Errata for reporting typos and errors in the course lecture notes and the slides.
- LLM Assignment 1 for discussing and asking questions about Assignment 1.
- LLM Assignment 2 for discussing and asking questions about Assignment 2.
- Find Assignment Partners for finding teammates for the course assignments.
If you feel like you would benefit from any other channel, feel free to suggest it to the teaching team!
We will prepare the course lecture notes as we go!
The individual chapters will be published in the course syllabus and updated throughout the semester.
Please report all errata to the teaching staff; we created an errata channel in
Other useful literature:
- Introduction to Natural Language Processing (Eisenstein)
- Deep Learning (Goodfellow, Bengio and Courville)
- AFLT Course Notes
Marks for the course will be determined by the following formula:
- 50% Final Exam
- 50% Assignments
On the Final Exam
The final exam is comprehensive and should be assumed to cover all the material in the slides and class notes.
On the Class Assignments
There will be 2 larger assignments in the course.
We require the solutions to be properly typeset.
We recommend using
markdown files with
MathJax for the mathematical expressions are also fine.