Methods in natural language processing (NLP) have matured to the point where they can address complex real-world problems. However, the process of advancing machine learning and NLP relies on the evaluation of constrained and often artificial tasks that may bear no clearly valid relationship to real-world problems. This disconnect leads to failures in generalization and limits methods’ utility.
In contrast, the social sciences provide a rich problem space, where questions of validity are at the center: what and how should we measure? Here, moving from language data to quantifiable social constructs demands complex reasoning over language.
The premise underpinning this talk is that an effective way to advance NLP as a field is to anchor it in the needs of social science. An emphasis on operational validity helps mitigate NLP’s benchmark myopia while also advancing the study of social phenomena. The talk will focus on contributions to two core activities within computational social science (CSS): the inductive development and measurement of latent constructs in text. Crucially, both are underpinned by human-centered validation; I will show how such an orientation led to a rethinking of standard topic model evaluation practices.
Alexander Hoyle is a PhD student in Computer Science at the University of Maryland, advised by Philip Resnik. His research is focused on the development and evaluation of NLP methods for computational social science. His work has appeared at conferences including ACL, EMNLP, NAACL, and NeurIPS. In the past, he has interned with the Fairness, Accountability, and Transparency group at Microsoft Research and the AllenNLP team at the Allen Institute for Artificial Intelligence.