AIConfPaper
← Search

Sora Takashima

3 accepted papers

2023

Pre-training Vision Transformers with Very Limited Synthesized Images

Ryo Nakamura, Hirokatsu Kataoka, Sora Takashima, Edgar Josafat Martinez Noriega, Rio Yokota, Nakamasa Inoue
ICCV 2023poster

Formula-driven supervised learning (FDSL) is a pre-training method that relies on synthetic images generated from mathematical formulae such as fractals. Prior work on FDSL has shown that pre-training vision transformers on such synthetic datasets can yield competitive accuracy on a wide range of d…

Cited by 10PDF ↗code ↗Scholar ↗
2023

Visual Atoms: Pre-Training Vision Transformers With Sinusoidal Waves

Sora Takashima, Ryo Hayamizu, Nakamasa Inoue, Hirokatsu Kataoka, Rio Yokota
CVPR 2023poster

Formula-driven supervised learning (FDSL) has been shown to be an effective method for pre-training vision transformers, where ExFractalDB-21k was shown to exceed the pre-training effect of ImageNet-21k. These studies also indicate that contours mattered more than textures when pre-training vision t…

Cited by 27Source ↗PDF ↗Scholar ↗
2022

Replacing Labeled Real-Image Datasets With Auto-Generated Contours

Hirokatsu Kataoka, Ryo Hayamizu, Ryosuke Yamada, Kodai Nakashima, Sora Takashima, Xinyu Zhang +3
CVPR 2022poster

In the present work, we show that the performance of formula-driven supervised learning (FDSL) can match or even exceed that of ImageNet-21k without the use of real images, human-, and self-supervision during the pre-training of Vision Transformers (ViTs). For example, ViT-Base pre-trained on ImageN…

Cited by 44PDF ↗Scholar ↗