Prompt Tuning or Fine-Tuning - Investigating Relational Knowledge in Pre-Trained Language Models
doi:10.24432/C5RC75
TL;DR
Achieving superior cloze-style fact extraction performance with adaptive fine-tuning on language models Extracting relational knowledge from large pre-trained language models by a cloze-style sentence serving as a query has shown promising results. In particular, language models can be queried similar to knowledge graphs. The performance of the relational fact extraction task depends significantly on the query sentence, also known under the term prompt. Tuning these prompts has shown to increase the precision on standard language models by a maximum of around 12% points. However, usually large amounts of data in the form of existing knowledge graph facts and large text corpora are needed to train the required additional model. In this work, we propose using a completely different approach: Instead of spending resources on training an additional model, we simply perform an adaptive fine-tuning of the pre-trained language model on the standard fill-mask task using a small training dataset of existing facts from a knowledge graph. We investigate the differences between complex prompting techniques and adaptive fine-tuning in an extensive evaluation. Remarkably, adaptive fine-tuning outperforms all baselines, even by using significantly fewer training facts. Additionally, we analyze the transfer learning capabilities of this adapted language model by training on a restricted set of relations to show that even fewer training relations are needed to achieve high knowledge extraction quality.
Citation
@inproceedings{
fichtel2021prompt,
title={Prompt Tuning or Fine-Tuning - Investigating Relational Knowledge in Pre-Trained Language Models},
author={Leandra Fichtel and Jan-Christoph Kalo and Wolf-Tilo Balke},
booktitle={3rd Conference on Automated Knowledge Base Construction},
year={2021},
url={https://openreview.net/forum?id=o7sMlpr9yBW},
doi={10.24432/C5RC75}
}