June 16, 2022 Natural Language Processing

BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese

  • 45 minutes
  • Nguyen Luong Tran, Duong Le, Dat Quoc Nguyen

  • InterSpeech 2022
Share

Abstract

We present BARTpho with two versions, BARTphosyllable and BARTphoword, which are the first public large-scale monolingual sequence-to-sequence models pre-trained for Vietnamese. BARTpho uses the “large” architecture and the pre-training scheme of the sequence-to-sequence denoising autoencoder BART, thus it is especially suitable for generative NLP tasks. We conduct experiments to compare our BARTpho with its competitor mBART on a downstream task of Vietnamese text summarization and show that: in both automatic and human evaluations, BARTpho outperforms the strong baseline mBART and improves the state-of-the-art. We further evaluate and compare BARTpho and mBART on the Vietnamese capitalization and punctuation restoration tasks and also find that BARTpho is more effective than mBART on these two tasks. We publicly release BARTpho to facilitate future research and applications of generative Vietnamese NLP tasks

Bibtex

@inproceedings{tran22b_interspeech,
  author={Nguyen Luong Tran and Duong Le and Dat Quoc Nguyen},
  title={{BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese}},
  year=2022,
  booktitle={Proc. Interspeech 2022},
  pages={1751--1755},
  doi={10.21437/Interspeech.2022-10177}
}
Back to Research
  • 45 minutes
  • Nguyen Luong Tran, Duong Le, Dat Quoc Nguyen

  • InterSpeech 2022
Share

Related publications