← Search

Szymon Antoniak

3 accepted papers

2024

Magnushammer: A Transformer-Based Approach to Premise Selection

ICLR 2024poster

This paper presents a novel approach to premise selection, a crucial reasoning task in automated theorem proving. Traditionally, symbolic methods that rely on extensive domain knowledge and engineering effort are applied to this task. In contrast, this work demonstrates that contrastive training wit…

Cited by 44SourcePDFScholar
2024

Mixture of Tokens: Continuous MoE through Cross-Example Aggregation

NeurIPS 2024poster

Mixture of Experts (MoE) models based on Transformer architecture are pushing the boundaries of language and vision tasks. The allure of these models lies in their ability to substantially increase the parameter count without a corresponding increase in FLOPs. Most widely adopted MoE models are disc…

2024

Scaling Laws for Fine-Grained Mixture of Experts

ICML 2024poster

Mixture of Experts (MoE) models have emerged as a primary solution for reducing the computational cost of Large Language Models. In this work, we analyze their scaling properties, highlighting certain arbitrary assumptions present in the existing literature. In particular, we introduce a new hyperpa…