Zachary Shinnick — accepted AI-conference papers

2026

Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformers

Zachary Shinnick, Liangze Jiang, Hemanth Saratchandran, Damien Teney, Anton van den Hengel

CVPR 2026

Transformers are remarkably versatile, suggesting the existence of generic inductive biases beneficial across modalities. In this work, we explore a new way to instil such biases in vision transformers (ViTs) through pretraining on procedurally generated data devoid of visual or semantic content. We

Cited by 0 Source ↗Scholar ↗

2026

Procedural Pretraining: Warming Up Language Models with Abstract Data

Liangze Jiang, Zachary Shinnick, Anton Hengel, Hemanth Saratchandran, Damien Teney

ICML 2026oral

Pretraining directly on web-scale corpora is the de facto paradigm for building language models. We study an alternative setting where the model is initially exposed to abstract structured data, as a means to ease the subsequent acquisition of rich semantic knowledge, much like humans learn simple l…

Cited by 0 Source ↗Scholar ↗