AIConfPaper
← Search

Ben Bogin

10 accepted papers

2025

Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code

Taishi Nakamura, Mayank Mishra, Simone Tedeschi, Yekun Chai, Jason T. Stillerman, Felix Friedrich +35
COLING 2025industry

Pretrained language models are integral part of AI applications, but their high computational cost for training limits accessibility. Initiatives such as Bloom and StarCoder aim to democratize access to pretrained models for collaborative community development. Despite these efforts, such models enc…

Cited by 2Source ↗PDF ↗Scholar ↗
2025

DataDecide: How to Predict Best Pretraining Data with Small Experiments

Ian Magnusson, Nguyen Tai, Ben Bogin, David Heineman, Jena D. Hwang, Luca Soldaini +7
ICML 2025poster

Because large language models are expensive to pretrain on different datasets, using smaller-scale experiments to decide on data is crucial for reducing costs. Which benchmarks and methods of making decisions from observed performance at small scale most accurately predict the datasets that yield th…

Cited by 0Source ↗PDF ↗Scholar ↗
2024

AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?

Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, Jonathan Berant
EMNLP 2024main

Language agents, built on top of language models (LMs), are systems that can interact with complex environments, such as the open web. In this work, we examine whether such agents can perform realistic and time-consuming tasks on the web, e.g., monitoring real-estate markets or locating relevant nea…

Cited by 13Source ↗PDF ↗Scholar ↗
2024

Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research

Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur +30
ACL 2024long

Information about pretraining corpora used to train the current best-performing language models is seldom discussed: commercial models rarely detail their data, and even open models are often released without accompanying training data or recipes to reproduce them. As a result, it is challenging to…

Cited by 156Source ↗PDF ↗code ↗Scholar ↗
2024

Leveraging Code to Improve In-Context Learning for Semantic Parsing

Ben Bogin, Shivanshu Gupta, Peter Clark, Ashish Sabharwal
NAACL 2024long

In-context learning (ICL) is an appealing approach for semantic parsing due to its few-shot nature and improved generalization. However, learning to parse to rare domain-specific languages (DSLs) from just a few demonstrations is challenging, limiting the performance of even the most capable LLMs.In…

Cited by 6Source ↗PDF ↗code ↗Scholar ↗
2024

SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories

Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark +2
EMNLP 2024main

Given that Large Language Models (LLMs) have made significant progress in writing code, can they now be used to autonomously reproduce results from research repositories? Such a capability would be a boon to the research community, helping researchers validate, understand, and extend prior work. To…

Cited by 9Source ↗PDF ↗code ↗Scholar ↗
2023

Answering Questions by Meta-Reasoning over Multiple Chains of Thought

Ori Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel Deutch, Jonathan Berant
EMNLP 2023long main

Modern systems for multi-hop question answering (QA) typically break questions into a sequence of reasoning steps, termed chain-of-thought (CoT), before arriving at a final answer. Often, multiple chains are sampled and aggregated through a voting mechanism over the final answers, but the intermedia…

Cited by 0Source ↗code ↗Scholar ↗
2023

Diverse Demonstrations Improve In-context Compositional Generalization

Itay Levy, Ben Bogin, Jonathan Berant
ACL 2023long

In-context learning has shown great success in i.i.d semantic parsing splits, where the training and test sets are drawn from the same distribution. In this setup, models are typically prompted with demonstrations that are similar to the input utterance. However, in the setup of compositional genera…

Cited by 136Source ↗PDF ↗code ↗Scholar ↗
2022

Unobserved Local Structures Make Compositional Generalization Hard

Ben Bogin, Shivanshu Gupta, Jonathan Berant
EMNLP 2022main

While recent work has shown that sequence-to-sequence models struggle to generalize to new compositions (termed compositional generalization), little is known on what makes compositional generalization hard on a particular test instance. In this work, we investigate the factors that make generalizat…

Cited by 27Source ↗PDF ↗code ↗Scholar ↗
2021

COVR: A Test-Bed for Visually Grounded Compositional Generalization with Real Images

Ben Bogin, Shivanshu Gupta, Matt Gardner, Jonathan Berant
EMNLP 2021main

While interest in models that generalize at test time to new compositions has risen in recent years, benchmarks in the visually-grounded domain have thus far been restricted to synthetic images. In this work, we propose COVR, a new test-bed for visually-grounded compositional generalization with rea…

Cited by 28Source ↗PDF ↗code ↗Scholar ↗