The Foundations of Tokenization: Statistical and Computational Concerns

Juan Luis Gastaldi, John Terilla, Luca Malagutti, Brian DuSell, Tim Vieira, Ryan Cotterell
January 2025