Sociolectal Analysis of Pretrained Language Models

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

Standard

Sociolectal Analysis of Pretrained Language Models. / Zhang, Sheng ; Zhang, Xin ; Zhang, Weiming ; Søgaard, Anders.

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021. p. 4581–4588.

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

Harvard

Zhang, S, Zhang, X, Zhang, W & Søgaard, A 2021, Sociolectal Analysis of Pretrained Language Models. in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp. 4581–4588, 2021 Conference on Empirical Methods in Natural Language Processing, 07/11/2021. https://doi.org/10.18653/v1/2021.emnlp-main.375

APA

Zhang, S., Zhang, X., Zhang, W., & Søgaard, A. (2021). Sociolectal Analysis of Pretrained Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 4581–4588). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.emnlp-main.375

Vancouver

Zhang S, Zhang X, Zhang W, Søgaard A. Sociolectal Analysis of Pretrained Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. 2021. p. 4581–4588 https://doi.org/10.18653/v1/2021.emnlp-main.375

Author

Zhang, Sheng ; Zhang, Xin ; Zhang, Weiming ; Søgaard, Anders. / Sociolectal Analysis of Pretrained Language Models. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021. pp. 4581–4588

Bibtex

@inproceedings{81d93e35f0764a68bd8848dfcdf0ca71,
title = "Sociolectal Analysis of Pretrained Language Models",
abstract = "Using data from English cloze tests, in which subjects also self-reported their gender, age, education, and race, we examine performance differences of pretrained language models across demographic groups, defined by these (protected) attributes. We demonstrate wide performance gaps across demographic groups and show that pretrained language models systematically disfavor young non-white male speakers; i.e., not only do pretrained language models learn social biases (stereotypical associations) – pretrained language models also learn sociolectal biases, learning to speak more like some than like others. We show, however, that, with the exception of BERT models, larger pretrained language models reduce some the performance gaps between majority and minority groups.",
author = "Sheng Zhang and Xin Zhang and Weiming Zhang and Anders S{\o}gaard",
year = "2021",
doi = "10.18653/v1/2021.emnlp-main.375",
language = "English",
pages = "4581–4588",
booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
publisher = "Association for Computational Linguistics",
note = "2021 Conference on Empirical Methods in Natural Language Processing ; Conference date: 07-11-2021 Through 11-11-2021",

}

RIS

TY - GEN

T1 - Sociolectal Analysis of Pretrained Language Models

AU - Zhang, Sheng

AU - Zhang, Xin

AU - Zhang, Weiming

AU - Søgaard, Anders

PY - 2021

Y1 - 2021

N2 - Using data from English cloze tests, in which subjects also self-reported their gender, age, education, and race, we examine performance differences of pretrained language models across demographic groups, defined by these (protected) attributes. We demonstrate wide performance gaps across demographic groups and show that pretrained language models systematically disfavor young non-white male speakers; i.e., not only do pretrained language models learn social biases (stereotypical associations) – pretrained language models also learn sociolectal biases, learning to speak more like some than like others. We show, however, that, with the exception of BERT models, larger pretrained language models reduce some the performance gaps between majority and minority groups.

AB - Using data from English cloze tests, in which subjects also self-reported their gender, age, education, and race, we examine performance differences of pretrained language models across demographic groups, defined by these (protected) attributes. We demonstrate wide performance gaps across demographic groups and show that pretrained language models systematically disfavor young non-white male speakers; i.e., not only do pretrained language models learn social biases (stereotypical associations) – pretrained language models also learn sociolectal biases, learning to speak more like some than like others. We show, however, that, with the exception of BERT models, larger pretrained language models reduce some the performance gaps between majority and minority groups.

U2 - 10.18653/v1/2021.emnlp-main.375

DO - 10.18653/v1/2021.emnlp-main.375

M3 - Article in proceedings

SP - 4581

EP - 4588

BT - Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

PB - Association for Computational Linguistics

T2 - 2021 Conference on Empirical Methods in Natural Language Processing

Y2 - 7 November 2021 through 11 November 2021

ER -

ID: 299822479