Natural Language Processing

ETH Zürich, Fall 2021: Course catalog

Course Description

This course presents topics in natural language processing with an emphasis on modern techniques, primarily focusing on statistical and deep learning approaches. The course provides an overview of the primary areas of research in language processing as well as a detailed exploration of the models and techniques used both in research and in commercial natural language processing systems.

The objective of the course is to learn the basic concepts in the statistical processing of natural languages. The course will be project-oriented so that the students can also gain hands-on experience with state-of-the-art tools and techniques.

Grading

Marks for the course will be determined by the following formula:
* 70% Final Exam
* 30% Course Project or Assignment

Lectures: Mon 12-14h Zoom (recurring link sent at start of semester). Recordings can be found in the password-protected course Polybox: https://polybox.ethz.ch/index.php/s/gplfKPSDUHSXDRq. The password can be found on the Moodle home page for the course.

Discussion Sections: Weds 12-14h HG F7; discussion sections will either be in person or via Zoom (same link as lecture), depending on the individual preferences of the teaching staff. Regardless, all sections will be recorded. Schedule to be posted at the beginning of the semester.

Textbooks: Introduction to Natural Language Processing (Eisenstein)
      Deep Learning (Goodfellow, Bengio and Courville)

News

18.09   Class website is online!
18.09   We are using Moodle as our discussion forum.
27.09   First lecture.
13.10   First discussion section.
03.10Project guidelines released.
31.10   Project proposals due.

Syllabus

Week Date   Topic Slides   Readings Supplementary Material Exercises Discussion Section
1 27.09.21 Introduction to Natural Language Lecture 1 Eisenstein Ch. 1
2 4.10.21 Backpropagation Lecture 2 Goodfellow, Bengio and Courville Ch. 6.5 Chris Olah's Blog
Justin Domke’s Notes
Tim Vieira’s Blog
Moritz Hardt’s Notes
Bauer (1974)
Baur and Strassen (1983)
Griewank and Walter (2008)
Eisner (2016)
Backpropagation Proof
Computation Graph for MLP
Computation Graph Example
Exercises
Solutions
13.10.21 (online)
3 11.10.21 Log-Linear Modeling---Meet the Softmax Lecture 3 Eisenstein Ch. 2 Ferraro and Eisner (2013)
Jason Eisner’s list of further resources on log-linear modeling
Exercises 20.10.21 (in person)
4 18.10.21 Sentiment Analysis with Multi-layer Perceptrons Lecture 4 Eisenstein Ch. 3 and Ch. 4
Goodfellow, Bengio and Courville Ch. 6
Wikipedia
Cybenko (1989)
Hanin and Selke (2018)
Pang and Lee (2008)
Iyyer et al. (2015)
word2vec Parameter Learning Explained
word2vec Explained
Exercises
Coding Exercises
27.10.21 (online)
5 25.10.21 Language Modeling with *n*-grams and LSTMs Lecture 5 (last semester) Eisenstein Ch. 6
Goodfellow, Bengio and Courville Ch. 10
Good Tutorial on n-gram smoothing
Good–Turing Smoothing
Kneser and Ney (1995)
Bengio et al. (2003)
Mikolov et al. (2010)
3.11.21 (online)
6 01.11.21 Part-of-Speech Tagging with CRFs Lecture 6 (last semester) Eisenstein Ch. 7 and 8 Tim Vieira's Blog
McCallum et al. (2000)
Lafferty et al. (2001)
Sutton and McCallum (2011)
Koller and Friedman (2009)
10.11.21 (online)
7 08.11.21 Context-Free Parsing with CKY Lecture 7 (last semester) Eisenstein Ch. 10 The Inside-Outside Algorithm
Jason Eisner’s Slides
Kasami (1966)
Younger (1967)
Cocke and Schwartz (1970)
17.11.21 (online)
8 15.11.21 Dependency Parsing with the Matrix-Tree Theorem Lecture 8 (last semester) Eisenstein Ch. 11 Koo et al. (2007)
Smith and Smith (2007)
McDonald and Satta (2007)
McDonald, Kübler and Nivre (2009)
24.11.21 (online)
9 22.11.21 Semantic Parsing with CCGs Lecture 9 (last semester) Eisenstein Ch. 9.3 and 12 Weir and Joshi (1988)
Kuhlmann and Satta (2014)
Mark Steedman's CCG slides
1.12.21 (in person)
10 29.11.21 Transliteration with WFSTs Lecture 10 (last semester) Eisenstein Ch. 9 Knight and Graehl (1998)
Mohri, Pereira and Riley (2008)
8.12.21 (online)
11 5.12.21 Machine Translation with Transformers Lecture 11 (last semester) Eisenstein Ch. 18 Neural Machine Translation
Vaswani et al. (2017)
Rush (2018)
15.12.21 (online)
12 13.12.21 Axes of Modeling Lecture 12 (last semester) Review: Eisenstein Ch. 2
Goodfellow, Bengio and Courville Ch. 5 and 11
22.12.21 (in person)
13 20.12.21 Bias and Fairness in NLP Lecture 13 (last semester) Bolukabasi et al. (2016)
Gonen and Goldberg (2019)
Hall Maudslay et al. (2019)
Vargas and Cotterell (2020)
A Course in Machine Learning Chapter 8

Weekly Exercises

We will release exercises every week that cover lecture material. Exercises will be released the day of the relevant lecture and reviewed in the discussion section the following week (i.e., 9 days later). These exercises are intended to give you the opportunity to test your understanding of the course material. They are not for a grade, nor will the TAs be able to offer individual feedback on your solutions. If you have questions regarding the exercises, we recommend bringing them to the relevant discussion section as the TAs will walk through related problems. Solutions will be released in the following week.

Course Project/Assignment

Every student has the option of completing either a research project or a structured assignment. This work will be worth 30% of your final mark. Both will be due on January 15th, 2022.

The research project is an open-ended project where students reimplement an existing research paper or perform novel research if they are so inclined. Projects can be completed in groups of up to 4. We will require you to write a 1-page project proposal where we will give you feedback on the idea (due October 31st). Submission can be done on the course Moodle page (one submission per team please). We will also require a progress report. Please send both the progress and final reports (including your recorded presentation) directly to your assigned TA. More details can be found in the project guidelines.

In the assignment, some of the questions will be more theoretical and resemble the questions you will see on the final exam. However, there will also be a substantial coding portion, which would not look like the exam questions. For instance, we may ask you to implement a recurrent neural dependency parser. Assignments must be completed individually, although you may discuss the assignment with other students. If you choose to do so, you must specify with whom you collaborated in your submission (see template below). We will release the assignment in two waves, corresponding to when course material is covered. Submission: Upload a zip file (with a single pdf containing all written answers to both parts of the assignment and the colab notebooks) to the assignment submission task on Moodle. For the writeup portion, we will only process a single pdf: if you include multiple pdfs in your submission, only one will be graded.

Materials

Contact

You can ask questions on Moodle through the Moodle Overflow forum. Please post questions there, so others can see them and share in the discussion. If you have questions which are not of general interest, please don’t hesitate to contact us directly, i.e., post a private note on Moodle or email Ryan with Clara cc-ed.