Systematic generalization is one of the most important open problems of neural networks: given a model trained to solve a certain problem, it will often fail on a test problem with different statistics than the training one, even if the problem should be solvable by the same algorithm. This indicates that the network relies on superficial statistics and memorization instead of learning algorithms. In contrast, the workhorse of human problem-solving is composition: it allows us to recombine solutions to known subproblems to solve problems we have never seen before. By introducing simple properties that seem essential for compositionality into the model architecture, we show dramatic improvements in the generalization ability of transformers.
Róbert Csordás is a PhD candidate at the Swiss AI lab IDSIA and currently doing an internship at DeepMind. His research interests are systematic generalizationi n the context of algorithmic reasoning and network architectures with nductive biases like information routing (attention, memory) and learning modular structures.