A Two-Sided Discussion of Preregistration of NLP Research

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

Standard

A Two-Sided Discussion of Preregistration of NLP Research. / Søgaard, Anders; Hershcovich, Daniel; de Lhoneux, Miryam.

EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference. Association for Computational Linguistics (ACL), 2023. p. 83-93.

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

Harvard

Søgaard, A, Hershcovich, D & de Lhoneux, M 2023, A Two-Sided Discussion of Preregistration of NLP Research. in EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference. Association for Computational Linguistics (ACL), pp. 83-93, 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, 02/05/2023. https://doi.org/10.18653/v1/2023.eacl-main.6

APA

Søgaard, A., Hershcovich, D., & de Lhoneux, M. (2023). A Two-Sided Discussion of Preregistration of NLP Research. In EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference (pp. 83-93). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2023.eacl-main.6

Vancouver

Søgaard A, Hershcovich D, de Lhoneux M. A Two-Sided Discussion of Preregistration of NLP Research. In EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference. Association for Computational Linguistics (ACL). 2023. p. 83-93 https://doi.org/10.18653/v1/2023.eacl-main.6

Author

Søgaard, Anders ; Hershcovich, Daniel ; de Lhoneux, Miryam. / A Two-Sided Discussion of Preregistration of NLP Research. EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference. Association for Computational Linguistics (ACL), 2023. pp. 83-93

Bibtex

@inproceedings{abbb276521ca4a7b98450f223caa549a,
title = "A Two-Sided Discussion of Preregistration of NLP Research",
abstract = "Van Miltenburg et al. (2021) suggest NLP research should adopt preregistration to prevent fishing expeditions and to promote publication of negative results. At face value, this is a very reasonable suggestion, seemingly solving many methodological problems with NLP research. We discuss pros and cons-some old, some new: a) Preregistration is challenged by the practice of retrieving hypotheses after the results are known; b) preregistration may bias NLP toward confirmatory research; c) preregistration must allow for reclassification of research as exploratory; d) preregistration may increase publication bias; e) preregistration may increase flag-planting; f) preregistration may increase p-hacking; and finally, g) preregistration may make us less risk tolerant. We cast our discussion as a dialogue, presenting both sides of the debate.",
author = "Anders S{\o}gaard and Daniel Hershcovich and {de Lhoneux}, Miryam",
note = "Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023 ; Conference date: 02-05-2023 Through 06-05-2023",
year = "2023",
doi = "10.18653/v1/2023.eacl-main.6",
language = "English",
pages = "83--93",
booktitle = "EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference",
publisher = "Association for Computational Linguistics (ACL)",
address = "United States",

}

RIS

TY - GEN

T1 - A Two-Sided Discussion of Preregistration of NLP Research

AU - Søgaard, Anders

AU - Hershcovich, Daniel

AU - de Lhoneux, Miryam

N1 - Publisher Copyright: © 2023 Association for Computational Linguistics.

PY - 2023

Y1 - 2023

N2 - Van Miltenburg et al. (2021) suggest NLP research should adopt preregistration to prevent fishing expeditions and to promote publication of negative results. At face value, this is a very reasonable suggestion, seemingly solving many methodological problems with NLP research. We discuss pros and cons-some old, some new: a) Preregistration is challenged by the practice of retrieving hypotheses after the results are known; b) preregistration may bias NLP toward confirmatory research; c) preregistration must allow for reclassification of research as exploratory; d) preregistration may increase publication bias; e) preregistration may increase flag-planting; f) preregistration may increase p-hacking; and finally, g) preregistration may make us less risk tolerant. We cast our discussion as a dialogue, presenting both sides of the debate.

AB - Van Miltenburg et al. (2021) suggest NLP research should adopt preregistration to prevent fishing expeditions and to promote publication of negative results. At face value, this is a very reasonable suggestion, seemingly solving many methodological problems with NLP research. We discuss pros and cons-some old, some new: a) Preregistration is challenged by the practice of retrieving hypotheses after the results are known; b) preregistration may bias NLP toward confirmatory research; c) preregistration must allow for reclassification of research as exploratory; d) preregistration may increase publication bias; e) preregistration may increase flag-planting; f) preregistration may increase p-hacking; and finally, g) preregistration may make us less risk tolerant. We cast our discussion as a dialogue, presenting both sides of the debate.

U2 - 10.18653/v1/2023.eacl-main.6

DO - 10.18653/v1/2023.eacl-main.6

M3 - Article in proceedings

AN - SCOPUS:85159579316

SP - 83

EP - 93

BT - EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference

PB - Association for Computational Linguistics (ACL)

T2 - 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023

Y2 - 2 May 2023 through 6 May 2023

ER -

ID: 356885601