Efficient methods for natural language processing: A survey

Marcos Treviso; Ji-Ung Lee; Tianchu Ji; Betty van Aken; Qingqing Cao; Manuel R Ciosici; Michael Hassid; Kenneth Heafield; Sara Hooker; Colin Raffel; Pedro H Martins; André FT Martins; Jessica Zosa Forde; Peter Milder; Edwin D. Simpson; Noam Slonim; Jesse Dodge; Emma Strubell; Niranjan Balasubramanian; Leon Derczynski; Iryna Gurevych; Roy Schwartz

doi:10.1162/tacl_a_00577

Efficient methods for natural language processing: A survey

Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H Martins, André FT Martins, Jessica Zosa Forde, Peter Milder, Edwin D. Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon DerczynskiIryna Gurevych, Roy Schwartz

School of Engineering Mathematics and Technology

Research output: Contribution to journal › Article (Academic Journal) › peer-review

49 Citations (Scopus)

Abstract

Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve performance means that resource consumption also grows. Such resources include data, time, storage, or energy, all of which are naturally limited and unevenly distributed. This motivates research into efficient methods that require fewer resources to achieve similar results. This survey synthesizes and relates current methods and findings in efficient NLP. We aim to provide both guidance for conducting NLP under limited resources, and point towards promising research directions for developing more efficient methods.

Original language	English
Pages (from-to)	826-860
Number of pages	35
Journal	Transactions of the Association for Computational Linguistics
Volume	11
DOIs	https://doi.org/10.1162/tacl_a_00577
Publication status	Published - 12 Jul 2023

Bibliographical note

Funding Information:
This work was initiated at and benefited substan-tially from the Dagstuhl Seminar 22232: Efficient and Equitable Natural Language Processing in the Age of Deep Learning. We further thank Yuki Arase, Jonathan Frankle, Alexander Koller, Alexander Loser, Alexandra Sasha Luccioni, Haritz Puerto, Nils Reimers, Leonardo Riberio, Anna Rogers, Andreas Ruckle, Noah A. Smith, and Thomas Wolf for a fruitful discussion and helpful feedback at the seminar. M.T. and A.M. acknowledge the European Research Council (ERC StG DeepSPIN 758969), EU’s Horizon Eu-rope Research and Innovation Actions (UTTER, contract 101070631), and Fundacao para a Ciencia e Tecnologia through contract UIDB/50008/2020. L.D. acknowledges support of the Independent Research Fund Denmark under project 9131-00131B, Verif-AI, and the Novo Nordisk Foundation project ClinRead, NNF19-OC0059138. Finally, we also thank the TACL reviewers and action editor for helpful discus¬sion and insightful feedback.

Funding Information:
This work was initiated at and benefited substantially from the Dagstuhl Seminar 22232: Efficient and Equitable Natural Language Processing in the Age of Deep Learning. We further thank Yuki Arase, Jonathan Frankle, Alexander Koller, Alexander Löser, Alexandra Sasha Luccioni, Haritz Puerto, Nils Reimers, Leonardo Riberio, Anna Rogers, Andreas Rücklé, Noah A. Smith, and Thomas Wolf for a fruitful discussion and helpful feedback at the seminar. M.T. and A.M. acknowledge the European Research Council (ERC StG DeepSPIN 758969), EU’s Horizon Europe Research and Innovation Actions (UTTER, contract 101070631), and Fundac¸ão para a Ciência e Tecnologia through contract UIDB/ 50008/2020. L.D. acknowledges support of the Independent Research Fund Denmark under project 9131-00131B, Verif-AI, and the Novo Nordisk Foundation project ClinRead, NNF19-OC0059138. Finally, we also thank the TACL reviewers and action editor for helpful discussion and insightful feedback.

Publisher Copyright:
© 2023 Association for Computational Linguistics.

Access to Document

10.1162/tacl_a_00577Licence: CC BY

Handle.net

Persistent link

Cite this

Treviso, M., Lee, J.-U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., Martins, P. H., Martins, A. FT., Zosa Forde, J., Milder, P., Simpson, E. D., Slonim, N., Dodge, J., Strubell, E., Balasubramanian, N., ... Schwartz, R. (2023). Efficient methods for natural language processing: A survey. Transactions of the Association for Computational Linguistics, 11, 826-860. https://doi.org/10.1162/tacl_a_00577

@article{61b15f118539421bafe6d71ad9f0b940,

title = "Efficient methods for natural language processing: A survey",

abstract = "Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve performance means that resource consumption also grows. Such resources include data, time, storage, or energy, all of which are naturally limited and unevenly distributed. This motivates research into efficient methods that require fewer resources to achieve similar results. This survey synthesizes and relates current methods and findings in efficient NLP. We aim to provide both guidance for conducting NLP under limited resources, and point towards promising research directions for developing more efficient methods.",

author = "Marcos Treviso and Ji-Ung Lee and Tianchu Ji and {van Aken}, Betty and Qingqing Cao and Ciosici, {Manuel R} and Michael Hassid and Kenneth Heafield and Sara Hooker and Colin Raffel and Martins, {Pedro H} and Martins, {Andr{\'e} FT} and {Zosa Forde}, Jessica and Peter Milder and Simpson, {Edwin D.} and Noam Slonim and Jesse Dodge and Emma Strubell and Niranjan Balasubramanian and Leon Derczynski and Iryna Gurevych and Roy Schwartz",

note = "Funding Information: This work was initiated at and benefited substan-tially from the Dagstuhl Seminar 22232: Efficient and Equitable Natural Language Processing in the Age of Deep Learning. We further thank Yuki Arase, Jonathan Frankle, Alexander Koller, Alexander Loser, Alexandra Sasha Luccioni, Haritz Puerto, Nils Reimers, Leonardo Riberio, Anna Rogers, Andreas Ruckle, Noah A. Smith, and Thomas Wolf for a fruitful discussion and helpful feedback at the seminar. M.T. and A.M. acknowledge the European Research Council (ERC StG DeepSPIN 758969), EU{\textquoteright}s Horizon Eu-rope Research and Innovation Actions (UTTER, contract 101070631), and Fundacao para a Ciencia e Tecnologia through contract UIDB/50008/2020. L.D. acknowledges support of the Independent Research Fund Denmark under project 9131-00131B, Verif-AI, and the Novo Nordisk Foundation project ClinRead, NNF19-OC0059138. Finally, we also thank the TACL reviewers and action editor for helpful discus¬sion and insightful feedback. Funding Information: This work was initiated at and benefited substantially from the Dagstuhl Seminar 22232: Efficient and Equitable Natural Language Processing in the Age of Deep Learning. We further thank Yuki Arase, Jonathan Frankle, Alexander Koller, Alexander L{\"o}ser, Alexandra Sasha Luccioni, Haritz Puerto, Nils Reimers, Leonardo Riberio, Anna Rogers, Andreas R{\"u}ckl{\'e}, Noah A. Smith, and Thomas Wolf for a fruitful discussion and helpful feedback at the seminar. M.T. and A.M. acknowledge the European Research Council (ERC StG DeepSPIN 758969), EU{\textquoteright}s Horizon Europe Research and Innovation Actions (UTTER, contract 101070631), and Fundac¸{\~a}o para a Ci{\^e}ncia e Tecnologia through contract UIDB/ 50008/2020. L.D. acknowledges support of the Independent Research Fund Denmark under project 9131-00131B, Verif-AI, and the Novo Nordisk Foundation project ClinRead, NNF19-OC0059138. Finally, we also thank the TACL reviewers and action editor for helpful discussion and insightful feedback. Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.",

year = "2023",

month = jul,

day = "12",

doi = "10.1162/tacl_a_00577",

language = "English",

volume = "11",

pages = "826--860",

journal = "Transactions of the Association for Computational Linguistics",

issn = "2307-387X",

publisher = "Massachusetts Institute of Technology Press",

}

Treviso, M, Lee, J-U, Ji, T, van Aken, B, Cao, Q, Ciosici, MR, Hassid, M, Heafield, K, Hooker, S, Raffel, C, Martins, PH, Martins, AFT, Zosa Forde, J, Milder, P, Simpson, ED, Slonim, N, Dodge, J, Strubell, E, Balasubramanian, N, Derczynski, L, Gurevych, I & Schwartz, R 2023, 'Efficient methods for natural language processing: A survey', Transactions of the Association for Computational Linguistics, vol. 11, pp. 826-860. https://doi.org/10.1162/tacl_a_00577

TY - JOUR

T1 - Efficient methods for natural language processing: A survey

AU - Treviso, Marcos

AU - Lee, Ji-Ung

AU - Ji, Tianchu

AU - van Aken, Betty

AU - Cao, Qingqing

AU - Ciosici, Manuel R

AU - Hassid, Michael

AU - Heafield, Kenneth

AU - Hooker, Sara

AU - Raffel, Colin

AU - Martins, Pedro H

AU - Martins, André FT

AU - Zosa Forde, Jessica

AU - Milder, Peter

AU - Simpson, Edwin D.

AU - Slonim, Noam

AU - Dodge, Jesse

AU - Strubell, Emma

AU - Balasubramanian, Niranjan

AU - Derczynski, Leon

AU - Gurevych, Iryna

AU - Schwartz, Roy

N1 - Funding Information: This work was initiated at and benefited substan-tially from the Dagstuhl Seminar 22232: Efficient and Equitable Natural Language Processing in the Age of Deep Learning. We further thank Yuki Arase, Jonathan Frankle, Alexander Koller, Alexander Loser, Alexandra Sasha Luccioni, Haritz Puerto, Nils Reimers, Leonardo Riberio, Anna Rogers, Andreas Ruckle, Noah A. Smith, and Thomas Wolf for a fruitful discussion and helpful feedback at the seminar. M.T. and A.M. acknowledge the European Research Council (ERC StG DeepSPIN 758969), EU’s Horizon Eu-rope Research and Innovation Actions (UTTER, contract 101070631), and Fundacao para a Ciencia e Tecnologia through contract UIDB/50008/2020. L.D. acknowledges support of the Independent Research Fund Denmark under project 9131-00131B, Verif-AI, and the Novo Nordisk Foundation project ClinRead, NNF19-OC0059138. Finally, we also thank the TACL reviewers and action editor for helpful discus¬sion and insightful feedback. Funding Information: This work was initiated at and benefited substantially from the Dagstuhl Seminar 22232: Efficient and Equitable Natural Language Processing in the Age of Deep Learning. We further thank Yuki Arase, Jonathan Frankle, Alexander Koller, Alexander Löser, Alexandra Sasha Luccioni, Haritz Puerto, Nils Reimers, Leonardo Riberio, Anna Rogers, Andreas Rücklé, Noah A. Smith, and Thomas Wolf for a fruitful discussion and helpful feedback at the seminar. M.T. and A.M. acknowledge the European Research Council (ERC StG DeepSPIN 758969), EU’s Horizon Europe Research and Innovation Actions (UTTER, contract 101070631), and Fundac¸ão para a Ciência e Tecnologia through contract UIDB/ 50008/2020. L.D. acknowledges support of the Independent Research Fund Denmark under project 9131-00131B, Verif-AI, and the Novo Nordisk Foundation project ClinRead, NNF19-OC0059138. Finally, we also thank the TACL reviewers and action editor for helpful discussion and insightful feedback. Publisher Copyright: © 2023 Association for Computational Linguistics.

PY - 2023/7/12

Y1 - 2023/7/12

N2 - Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve performance means that resource consumption also grows. Such resources include data, time, storage, or energy, all of which are naturally limited and unevenly distributed. This motivates research into efficient methods that require fewer resources to achieve similar results. This survey synthesizes and relates current methods and findings in efficient NLP. We aim to provide both guidance for conducting NLP under limited resources, and point towards promising research directions for developing more efficient methods.

AB - Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve performance means that resource consumption also grows. Such resources include data, time, storage, or energy, all of which are naturally limited and unevenly distributed. This motivates research into efficient methods that require fewer resources to achieve similar results. This survey synthesizes and relates current methods and findings in efficient NLP. We aim to provide both guidance for conducting NLP under limited resources, and point towards promising research directions for developing more efficient methods.

U2 - 10.1162/tacl_a_00577

DO - 10.1162/tacl_a_00577

M3 - Article (Academic Journal)

SN - 2307-387X

VL - 11

SP - 826

EP - 860

JO - Transactions of the Association for Computational Linguistics

JF - Transactions of the Association for Computational Linguistics

ER -

Efficient methods for natural language processing: A survey

Abstract

Bibliographical note

Access to Document

Handle.net

Fingerprint

Cite this