Mun Wai Lee

PaCa-ViT: Learning Patch-to-Cluster Attention in Vision Transformers

Vision Transformers (ViTs) are built on the assumption of treating image patches as "visual tokens" and learn patch-to-patch attention. The patch embedding based tokenizer has a semantic gap with respect to its counterpart, the textual tokenizer. The patch-to-patch attention suffers from the quadrat…