RoBERTa: A Robustly Optimized BERT Pretraining Approach

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

PDF Abstract

Results from the Paper


 Ranked #1 on Only Connect Walls Dataset Task 1 (Grouping) on OCW (Wasserstein Distance (WD) metric, using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Natural Language Inference ANLI test RoBERTa (Large) A1 72.4 # 5
A2 49.8 # 14
A3 44.4 # 16
Text Classification arXiv-10 RoBERTa Accuracy 0.779 # 2
Stock Market Prediction Astock RoBERTa WWM Ext (News+Factors) Accuray 62.49 # 11
F1-score 62.54 # 11
Recall 62.51 # 11
Precision 62.59 # 11
Stock Market Prediction Astock RoBERTa WWM Ext (News) Accuray 61.34 # 12
F1-score 61.48 # 12
Recall 61.32 # 12
Precision 61.97 # 12
Linguistic Acceptability CoLA RoBERTa (ensemble) Accuracy 67.8% # 20
Sentence Completion HellaSwag RoBERTa-Large 355M Accuracy 81.7 # 36
Type prediction ManyTypes4TypeScript RoBERTa Average Accuracy 59.84 # 7
Average Precision 57.45 # 5
Average Recall 57.62 # 5
Average F1 57.54 # 5
Multi-task Language Understanding MMLU RoBERTa-base 125M (fine-tuned) Average (%) 27.9 # 94
Semantic Textual Similarity MRPC RoBERTa (ensemble) Accuracy 92.3% # 3
Natural Language Inference MultiNLI RoBERTa Matched 90.8 # 8
Natural Language Inference MultiNLI RoBERTa (ensemble) Mismatched 90.2 # 7
Only Connect Walls Dataset Task 1 (Grouping) OCW RoBERTa (LARGE) # Correct Groups 29 ± 3 # 21
Fowlkes Mallows Score (FMS) 26.7 ± .2 # 19
Adjusted Rand Index (ARI) 8.4 ± .3 # 19
Adjusted Mutual Information (AMI) 9.4 ± .4 # 20
# Solved Walls 0 ± 0 # 10
Wasserstein Distance (WD) 88.4 ± .4 # 1
Natural Language Inference QNLI RoBERTa (ensemble) Accuracy 98.9% # 5
Question Answering Quora Question Pairs RoBERTa Accuracy 90.2% # 6
Reading Comprehension RACE RoBERTa Accuracy 83.2 # 7
Accuracy (High) 81.3 # 6
Accuracy (Middle) 86.5 # 6
Natural Language Inference RTE RoBERTa Accuracy 88.2% # 18
Natural Language Inference RTE RoBERTa (ensemble) Accuracy 88.2% # 18
Question Answering SQuAD2.0 RoBERTa (single model) EM 86.820 # 90
F1 89.795 # 91
Question Answering SQuAD2.0 dev RoBERTa (no data aug) F1 89.4 # 3
EM 86.5 # 3
Sentiment Analysis SST-2 Binary classification RoBERTa (ensemble) Accuracy 96.7 # 11
Semantic Textual Similarity STS Benchmark RoBERTa Pearson Correlation 0.922 # 7
Common Sense Reasoning SWAG RoBERTa Test 89.9 # 2
Natural Language Inference WNLI RoBERTa (ensemble) Accuracy 89 # 8

Results from Other Papers


Task Dataset Model Metric Name Metric Value Rank Uses Extra
Training Data
Source Paper Compare
Common Sense Reasoning CommonsenseQA RoBERTa-Large 355M Accuracy 72.1 # 18
Sentence Completion HellaSwag RoBERTa-Large Ensemble Accuracy 85.5 # 19
Question Answering PIQA RoBERTa-Large 355M Accuracy 79.4 # 32
Question Answering SIQA RoBERTa-Large 355M (fine-tuned) Accuracy 76.7 # 8
Document Image Classification RVL-CDIP Roberta base Accuracy 90.06 # 30
Parameters 125M # 19

Methods