PGTask: Introducing the Task of Profile Generation from Dialogues

13 Apr 2023  ·  Rui Ribeiro, Joao P. Carvalho, Luísa Coheur ·

Recent approaches have attempted to personalize dialogue systems by leveraging profile information into models. However, this knowledge is scarce and difficult to obtain, which makes the extraction/generation of profile information from dialogues a fundamental asset. To surpass this limitation, we introduce the Profile Generation Task (PGTask). We contribute with a new dataset for this problem, comprising profile sentences aligned with related utterances, extracted from a corpus of dialogues. Furthermore, using state-of-the-art methods, we provide a benchmark for profile generation on this novel dataset. Our experiments disclose the challenges of profile generation, and we hope that this introduces a new research direction.

PDF Abstract

Datasets


Introduced in the Paper:

PGDataset

Used in the Paper:

MultiNLI

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Profile Generation PGDataset gpt2-small BLEU-1 61.30 # 1
BLEU-2 32.3 # 1
BLEU-3 20.62 # 1
BLEU-4 9.44 # 1
ROUGE-1 50.07 # 1
ROUGE-2 28.31 # 1
ROUGE-L 50.00 # 1
BertScore 94.39 # 2
Profile Generation PGDataset gpt2-medium BLEU-1 59.31 # 2
BLEU-2 25.94 # 2
BLEU-3 15.3 # 2
BLEU-4 9.17 # 2
ROUGE-1 46.32 # 2
ROUGE-2 24.14 # 2
ROUGE-L 45.88 # 2
BertScore 94.76 # 1

Methods