Findings of the WMT 2020 Shared Task on Machine Translation Robustness

Lucia Specia; Zhenhao Li; Juan Pino; Vishrav Chaudhary; Francisco Guzmán; Paul Michel; Graham Neubig; Hassan Sajjad; Nadir Durrani; Yonatan Belinkov; Philipp Koehn; Xian Li

Findings of the WMT 2020 Shared Task on Machine Translation Robustness

Lucia Specia, Zhenhao Li, Juan Pino, Vishrav Chaudhary, Francisco Guzmán, Paul Michel, Graham Neubig, Hassan Sajjad, Nadir Durrani, Yonatan Belinkov, Philipp Koehn, Xian Li

Qatar Computing Research Institute

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

20 Citations (Scopus)

Abstract

We report the findings of the second edition of the shared task on improving robustness in Machine Translation (MT). The task aims to test current machine translation systems in their ability to handle challenges facing MT models to be deployed in the real world, including domain diversity and non-standard texts common in user generated content, especially in social media. We cover two language pairs - English-German and English-Japanese and provide test sets in zero-shot and few-shot variants. Participating systems are evaluated both automatically and manually, with an additional human evaluation for “catastrophic errors”. We received 59 submissions by 11 participating teams from a variety of types of institutions.

Original language	English
Title of host publication	5th Conference on Machine Translation, WMT 2020 - Proceedings
Editors	Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-Jussa, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Andre Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri
Publisher	Association for Computational Linguistics (ACL)
Pages	76-91
Number of pages	16
ISBN (Electronic)	9781948087810
Publication status	Published - 2020
Event	5th Conference on Machine Translation, WMT 2020 - Virtual, Online Duration: 19 Nov 2020 → 20 Nov 2020

Publication series

Name	5th Conference on Machine Translation, WMT 2020 - Proceedings

Conference

Conference	5th Conference on Machine Translation, WMT 2020
City	Virtual, Online
Period	19/11/20 → 20/11/20

Cite this

Specia, L., Li, Z., Pino, J., Chaudhary, V., Guzmán, F., Michel, P., Neubig, G., Sajjad, H., Durrani, N., Belinkov, Y., Koehn, P., & Li, X. (2020). Findings of the WMT 2020 Shared Task on Machine Translation Robustness. In L. Barrault, O. Bojar, F. Bougares, R. Chatterjee, M. R. Costa-Jussa, C. Federmann, M. Fishel, A. Fraser, Y. Graham, P. Guzman, B. Haddow, M. Huck, A. J. Yepes, P. Koehn, A. Martins, M. Morishita, C. Monz, M. Nagata, T. Nakazawa, & M. Negri (Eds.), 5th Conference on Machine Translation, WMT 2020 - Proceedings (pp. 76-91). (5th Conference on Machine Translation, WMT 2020 - Proceedings). Association for Computational Linguistics (ACL).

Specia, Lucia ; Li, Zhenhao ; Pino, Juan et al. / Findings of the WMT 2020 Shared Task on Machine Translation Robustness. 5th Conference on Machine Translation, WMT 2020 - Proceedings. editor / Loic Barrault ; Ondrej Bojar ; Fethi Bougares ; Rajen Chatterjee ; Marta R. Costa-Jussa ; Christian Federmann ; Mark Fishel ; Alexander Fraser ; Yvette Graham ; Paco Guzman ; Barry Haddow ; Matthias Huck ; Antonio Jimeno Yepes ; Philipp Koehn ; Andre Martins ; Makoto Morishita ; Christof Monz ; Masaaki Nagata ; Toshiaki Nakazawa ; Matteo Negri. Association for Computational Linguistics (ACL), 2020. pp. 76-91 (5th Conference on Machine Translation, WMT 2020 - Proceedings).

@inproceedings{3802be5376754d33821674b96ce81a70,

title = "Findings of the WMT 2020 Shared Task on Machine Translation Robustness",

abstract = "We report the findings of the second edition of the shared task on improving robustness in Machine Translation (MT). The task aims to test current machine translation systems in their ability to handle challenges facing MT models to be deployed in the real world, including domain diversity and non-standard texts common in user generated content, especially in social media. We cover two language pairs - English-German and English-Japanese and provide test sets in zero-shot and few-shot variants. Participating systems are evaluated both automatically and manually, with an additional human evaluation for “catastrophic errors”. We received 59 submissions by 11 participating teams from a variety of types of institutions.",

author = "Lucia Specia and Zhenhao Li and Juan Pino and Vishrav Chaudhary and Francisco Guzm{\'a}n and Paul Michel and Graham Neubig and Hassan Sajjad and Nadir Durrani and Yonatan Belinkov and Philipp Koehn and Xian Li",

note = "Publisher Copyright: {\textcopyright} 2020 Association for Computational Linguistics; 5th Conference on Machine Translation, WMT 2020 ; Conference date: 19-11-2020 Through 20-11-2020",

year = "2020",

language = "English",

series = "5th Conference on Machine Translation, WMT 2020 - Proceedings",

publisher = "Association for Computational Linguistics (ACL)",

pages = "76--91",

editor = "Loic Barrault and Ondrej Bojar and Fethi Bougares and Rajen Chatterjee and Costa-Jussa, {Marta R.} and Christian Federmann and Mark Fishel and Alexander Fraser and Yvette Graham and Paco Guzman and Barry Haddow and Matthias Huck and Yepes, {Antonio Jimeno} and Philipp Koehn and Andre Martins and Makoto Morishita and Christof Monz and Masaaki Nagata and Toshiaki Nakazawa and Matteo Negri",

booktitle = "5th Conference on Machine Translation, WMT 2020 - Proceedings",

address = "United States",

}

Specia, L, Li, Z, Pino, J, Chaudhary, V, Guzmán, F, Michel, P, Neubig, G, Sajjad, H, Durrani, N, Belinkov, Y, Koehn, P & Li, X 2020, Findings of the WMT 2020 Shared Task on Machine Translation Robustness. in L Barrault, O Bojar, F Bougares, R Chatterjee, MR Costa-Jussa, C Federmann, M Fishel, A Fraser, Y Graham, P Guzman, B Haddow, M Huck, AJ Yepes, P Koehn, A Martins, M Morishita, C Monz, M Nagata, T Nakazawa & M Negri (eds), 5th Conference on Machine Translation, WMT 2020 - Proceedings. 5th Conference on Machine Translation, WMT 2020 - Proceedings, Association for Computational Linguistics (ACL), pp. 76-91, 5th Conference on Machine Translation, WMT 2020, Virtual, Online, 19/11/20.

Findings of the WMT 2020 Shared Task on Machine Translation Robustness. / Specia, Lucia; Li, Zhenhao; Pino, Juan et al.
5th Conference on Machine Translation, WMT 2020 - Proceedings. ed. / Loic Barrault; Ondrej Bojar; Fethi Bougares; Rajen Chatterjee; Marta R. Costa-Jussa; Christian Federmann; Mark Fishel; Alexander Fraser; Yvette Graham; Paco Guzman; Barry Haddow; Matthias Huck; Antonio Jimeno Yepes; Philipp Koehn; Andre Martins; Makoto Morishita; Christof Monz; Masaaki Nagata; Toshiaki Nakazawa; Matteo Negri. Association for Computational Linguistics (ACL), 2020. p. 76-91 (5th Conference on Machine Translation, WMT 2020 - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Findings of the WMT 2020 Shared Task on Machine Translation Robustness

AU - Specia, Lucia

AU - Li, Zhenhao

AU - Pino, Juan

AU - Chaudhary, Vishrav

AU - Guzmán, Francisco

AU - Michel, Paul

AU - Neubig, Graham

AU - Sajjad, Hassan

AU - Durrani, Nadir

AU - Belinkov, Yonatan

AU - Koehn, Philipp

AU - Li, Xian

PY - 2020

Y1 - 2020

N2 - We report the findings of the second edition of the shared task on improving robustness in Machine Translation (MT). The task aims to test current machine translation systems in their ability to handle challenges facing MT models to be deployed in the real world, including domain diversity and non-standard texts common in user generated content, especially in social media. We cover two language pairs - English-German and English-Japanese and provide test sets in zero-shot and few-shot variants. Participating systems are evaluated both automatically and manually, with an additional human evaluation for “catastrophic errors”. We received 59 submissions by 11 participating teams from a variety of types of institutions.

AB - We report the findings of the second edition of the shared task on improving robustness in Machine Translation (MT). The task aims to test current machine translation systems in their ability to handle challenges facing MT models to be deployed in the real world, including domain diversity and non-standard texts common in user generated content, especially in social media. We cover two language pairs - English-German and English-Japanese and provide test sets in zero-shot and few-shot variants. Participating systems are evaluated both automatically and manually, with an additional human evaluation for “catastrophic errors”. We received 59 submissions by 11 participating teams from a variety of types of institutions.

UR - http://www.scopus.com/inward/record.url?scp=85123824869&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85123824869

T3 - 5th Conference on Machine Translation, WMT 2020 - Proceedings

SP - 76

EP - 91

BT - 5th Conference on Machine Translation, WMT 2020 - Proceedings

A2 - Barrault, Loic

A2 - Bojar, Ondrej

A2 - Bougares, Fethi

A2 - Chatterjee, Rajen

A2 - Costa-Jussa, Marta R.

A2 - Federmann, Christian

A2 - Fishel, Mark

A2 - Fraser, Alexander

A2 - Graham, Yvette

A2 - Guzman, Paco

A2 - Haddow, Barry

A2 - Huck, Matthias

A2 - Yepes, Antonio Jimeno

A2 - Koehn, Philipp

A2 - Martins, Andre

A2 - Morishita, Makoto

A2 - Monz, Christof

A2 - Nagata, Masaaki

A2 - Nakazawa, Toshiaki

A2 - Negri, Matteo

PB - Association for Computational Linguistics (ACL)

T2 - 5th Conference on Machine Translation, WMT 2020

Y2 - 19 November 2020 through 20 November 2020

ER -

Specia L, Li Z, Pino J, Chaudhary V, Guzmán F, Michel P et al. Findings of the WMT 2020 Shared Task on Machine Translation Robustness. In Barrault L, Bojar O, Bougares F, Chatterjee R, Costa-Jussa MR, Federmann C, Fishel M, Fraser A, Graham Y, Guzman P, Haddow B, Huck M, Yepes AJ, Koehn P, Martins A, Morishita M, Monz C, Nagata M, Nakazawa T, Negri M, editors, 5th Conference on Machine Translation, WMT 2020 - Proceedings. Association for Computational Linguistics (ACL). 2020. p. 76-91. (5th Conference on Machine Translation, WMT 2020 - Proceedings).

Findings of the WMT 2020 Shared Task on Machine Translation Robustness

Abstract

Publication series

Conference

Other files and links

Fingerprint

Cite this