AIConfPaper
← Search

Johannes Welbl

6 accepted papers

2022

An empirical analysis of compute-optimal large language model training

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford +16
NeurIPS 2022accept

We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of trai…

Cited by 171Source ↗PDF ↗Scholar ↗
2022

Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models

Maribeth Rauh, John F J Mellor, Jonathan Uesato, Po-Sen Huang, Johannes Welbl, Laura Weidinger +6
NeurIPS 2022accept

Large language models produce human-like text that drive a growing number of applications. However, recent literature and, increasingly, real world observations, have demonstrated that these models can generate language that is toxic, biased, untruthful or otherwise harmful. Though work to evalua…

Cited by 55Source ↗PDF ↗Scholar ↗
2021

Challenges in Detoxifying Language Models

Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks +4
EMNLP 2021finding

Large language models (LM) generate remarkably fluent text and can be efficiently adapted across NLP tasks. Measuring and guaranteeing the quality of generated text in terms of safety is imperative for deploying LMs in the real world; to this end, prior work often relies on automatic evaluation of L…

Cited by 256Source ↗PDF ↗Scholar ↗
2020

Towards Verified Robustness under Text Deletion Interventions

Johannes Welbl, Po-Sen Huang, Robert Stanforth, Sven Gowal, Krishnamurthy (Dj) Dvijotham, Martin Szummer +1
ICLR 2020poster

Neural networks are widely used in Natural Language Processing, yet despite their empirical successes, their behaviour is brittle: they are both over-sensitive to small input changes, and under-sensitive to deletions of large fractions of input text. This paper aims to tackle under-sensitivity in th…

Cited by 3Source ↗Scholar ↗
2017

Frustratingly Short Attention Spans in Neural Language Modeling

Michał Daniluk, Tim Rocktäschel, Johannes Welbl, Sebastian Riedel
ICLR 2017poster

Current language modeling architectures often use recurrent neural networks. Recently, various methods for incorporating differentiable memory into these architectures have been proposed. When predicting the next token, these models query information from a memory of the recent history and thus can…

Cited by 157Source ↗Scholar ↗
2016

Complex Embeddings for Simple Link Prediction

Théo Trouillon, Johannes Welbl, Sebastian Riedel, Eric Gaussier, Guillaume Bouchard
ICML 2016poster

In statistical relational learning, the link prediction problem is key to automatically understand the structure of large knowledge bases. As in previous studies, we propose to solve this problem through latent factorization. However, here we make use of complex valued embeddings. The composition of…

Cited by 4125Source ↗PDF ↗code ↗Scholar ↗