Boris Katz — accepted AI-conference papers

2025

Population Transformer: Learning Population-level Representations of Neural Activity

Geeling Chau, Christopher Wang, Sabera J Talukder, Vighnesh Subramaniam, Saraswati Soedarmadji, Yisong Yue +2

ICLR 2025oral

We present a self-supervised framework that learns population-level codes for arbitrary ensembles of neural recordings at scale. We address key challenges in scaling models with neural time-series data, namely, sparse and variable electrode distribution across subjects and datasets. The Population T…

Cited by 3 Source ↗PDF ↗code ↗Scholar ↗

2025

Training the Untrainable: Introducing Inductive Bias via Representational Alignment

Vighnesh Subramaniam, David Mayo, Colin Conwell, Tomaso Poggio, Boris Katz, Brian Cheung +1

NeurIPS 2025poster

We demonstrate that architectures which traditionally are considered to be ill-suited for a task can be trained using inductive biases from another architecture. We call a network untrainable when it overfits, underfits, or converges to poor results even when tuning their hyperparameters. For examp…

Cited by 0 Source ↗Scholar ↗

2024

Brain Treebank: Large-scale intracranial recordings from naturalistic language stimuli

Christopher Wang, Adam Uri Yaari, Aaditya K Singh, Vighnesh Subramaniam, Dana Rosenfarb, Jan DeWitt +7

NeurIPS 2024oral

We present the Brain Treebank, a large-scale dataset of electrophysiological neural responses, recorded from intracranial probes while 10 subjects watched one or more Hollywood movies. Subjects watched on average 2.6 Hollywood movies, for an average viewing time of 4.3 hours, and a total of 43 hours…

Cited by 3 Source ↗PDF ↗Scholar ↗

2024

BrainBits: How Much of the Brain are Generative Reconstruction Methods Using?

David Mayo, Christopher Wang, Asa Harbin, Abdulrahman Alabdulkareem, Albert Eaton Shaw, Boris Katz +1

NeurIPS 2024poster

When evaluating stimuli reconstruction results it is tempting to assume that higher fidelity text and image generation is due to an improved understanding of the brain or more powerful signal extraction from neural recordings. However, in practice, new reconstruction methods could improve performan…

Cited by 0 Source ↗PDF ↗Scholar ↗

2024

Revealing Vision-Language Integration in the Brain with Multimodal Networks

Vighnesh Subramaniam, Colin Conwell, Christopher Wang, Gabriel Kreiman, Boris Katz, Ignacio Cases +1

ICML 2024poster

We use (multi)modal deep neural networks (DNNs) to probe for sites of multimodal integration in the human brain by predicting stereoencephalography (SEEG) recordings taken while human subjects watched movies. We operationalize sites of multimodal integration as regions where a multimodal vision-lang…

Cited by 7 Source ↗PDF ↗code ↗Scholar ↗

2023

BrainBERT: Self-supervised representation learning for intracranial recordings

Christopher Wang, Vighnesh Subramaniam, Adam Uri Yaari, Gabriel Kreiman, Boris Katz, Ignacio Cases +1

ICLR 2023poster

We create a reusable Transformer, BrainBERT, for intracranial recordings bringing modern representation learning approaches to neuroscience. Much like in NLP and speech recognition, this Transformer enables classifying complex concepts, i.e., decoding neural data, with higher accuracy and with much…

Cited by 39 Source ↗PDF ↗code ↗Scholar ↗

2023

How hard are computer vision datasets? Calibrating dataset difficulty to viewing time

David Mayo, Jesse Cummings, Xinyu Lin, Dan Gutfreund, Boris Katz, Andrei Barbu

NeurIPS 2023poster

Humans outperform object recognizers despite the fact that models perform well on current datasets, including those explicitly designed to challenge machines with debiased images or distribution shift. This problem persists, in part, because we have no guidance on the absolute difficulty of an image…

Cited by 19 Source ↗PDF ↗Scholar ↗

2023

Zero-Shot Linear Combinations of Grounded Social Interactions with Linear Social MDPs

Ravi Tejwani, Yen-Ling Kuo, Tianmin Shu, Bennett Stankovits, Dan Gutfreund, Joshua B. Tenenbaum +2

AAAI 2023technical

Humans and animals engage in rich social interactions. It is often theorized that a relatively small number of basic social interactions give rise to the full range of behavior observed. But no computational theory explaining how social interactions combine together has been proposed before. We do s…

Cited by 1 Source ↗PDF ↗Scholar ↗

2022

Incorporating Rich Social Interactions Into MDPs

Ravi Tejwani, Yen-Ling Kuo, Tianmin Shu, Bennett Stankovits, Dan Gutfreund, Joshua B. Tenenbaum +2

ICRA 2022poster

Much of what we do as humans is engage socially with other agents, a skill that robots must also eventually possess. We demonstrate that a rich theory of social interactions originating from microsociology can be formalized by extending a nested MDP where agents reason about arbitrary functions of e…

Cited by 10 Source ↗Scholar ↗

2022

The Aligned Multimodal Movie Treebank: An audio, video, dependency-parse treebank

Adam Yaari, Jan DeWitt, Henry Hu, Bennett Stankovits, Sue Felshin, Yevgeni Berzak +4

EMNLP 2022main

Treebanks have traditionally included only text and were derived from written sources such as newspapers or the web. We introduce the Aligned Multimodal Movie Treebank (AMMT), an English language treebank derived from dialog in Hollywood movies which includes transcriptions of the audio-visual strea…

Cited by 7 Source ↗PDF ↗code ↗Scholar ↗

2022

Trajectory Prediction with Linguistic Representations

Yen-Ling Kuo, Xin Huang, Andrei Barbu, Stephen G. McGill, Boris Katz, John J. Leonard +1

ICRA 2022poster

Language allows humans to build mental models that interpret what is happening around them resulting in more accurate long-term predictions. We present a novel trajectory prediction model that uses linguistic intermediate representations to forecast trajectories, and is trained using trajectory samp…

Cited by 22 Source ↗Scholar ↗

2021

Compositional Networks Enable Systematic Generalization for Grounded Language Understanding

Yen-Ling Kuo, Boris Katz, Andrei Barbu

EMNLP 2021finding

Humans are remarkably flexible when understanding new sentences that include combinations of concepts they have never encountered before. Recent work has shown that while deep networks can mimic some human language abilities when presented with novel sentences, systematic variation uncovers the limi…

Cited by 25 Source ↗PDF ↗code ↗Scholar ↗

2021

Measuring Social Biases in Grounded Vision and Language Embeddings

Candace Ross, Boris Katz, Andrei Barbu

NAACL 2021long

We generalize the notion of measuring social biases in word embeddings to visually grounded word embeddings. Biases are present in grounded embeddings, and indeed seem to be equally or more significant than for ungrounded embeddings. This is despite the fact that vision and language can suffer from…

Cited by 76 Source ↗PDF ↗code ↗Scholar ↗

2021

Multi-resolution modeling of a discrete stochastic process identifies causes of cancer

Adam Uri Yaari, Maxwell Sherman, Oliver Clarke Priebe, Po-Ru Loh, Boris Katz, Andrei Barbu +1

ICLR 2021poster

Detection of cancer-causing mutations within the vast and mostly unexplored human genome is a major challenge. Doing so requires modeling the background mutation rate, a highly non-stationary stochastic process, across regions of interest varying in size from one to millions of positions. Here, we p…

Cited by 3 Source ↗PDF ↗Scholar ↗

2021

Neural Regression, Representational Similarity, Model Zoology & Neural Taskonomy at Scale in Rodent Visual Cortex

Colin Conwell, David Mayo, Andrei Barbu, Michael A Buice, George A. Alvarez, Boris Katz

NeurIPS 2021poster

How well do deep neural networks fare as models of mouse visual cortex? A majority of research to date suggests results far more mixed than those produced in the modeling of primate visual cortex. Here, we perform a large-scale benchmarking of dozens of deep neural network models in mouse visual cor…

Cited by 31 Source ↗PDF ↗code ↗Scholar ↗

2021

PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception

Aviv Netanyahu, Tianmin Shu, Boris Katz, Andrei Barbu, Joshua B. Tenenbaum

AAAI 2021technical

The ability to perceive and reason about social interactions in the context of physical environments is core to human social intelligence and human-machine cooperation. However, no prior dataset or benchmark has systematically evaluated physically grounded perception of complex social interactions t…

Cited by 36 Source ↗PDF ↗Scholar ↗

2021

Social Interactions as Recursive MDPs

Ravi Tejwani, Yen-Ling Kuo, Tianmin Shu, Boris Katz, Andrei Barbu

CoRL 2021poster

While machines and robots must interact with humans, providing them with social skills has been a largely overlooked topic. This is mostly a consequence of the fact that tasks such as navigation, command following, and even game playing are well-defined, while social reasoning still mostly remains a…

Cited by 26 Source ↗Scholar ↗

2020

Deep compositional robotic planners that follow natural language commands

Yen-Ling Kuo, Boris Katz, Andrei Barbu

ICRA 2020poster

We demonstrate how a sampling-based robotic planner can be augmented to learn to understand a sequence of natural language commands in a continuous configuration space to move and manipulate objects. Our approach combines a deep network structured according to the parse of a complex command that inc…

Cited by 29 Source ↗Scholar ↗

2020

Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulas

Yen-Ling Kuo, Boris Katz, Andrei Barbu

IROS 2020poster

We demonstrate a reinforcement learning agent which uses a compositional recurrent neural network that takes as input an LTL formula and determines satisfying actions. The input LTL formulas have never been seen before, yet the network performs zero-shot generalization to satisfy them. This is a nov…

Cited by 57 Source ↗code ↗Scholar ↗

2020

Learning a natural-language to LTL executable semantic parser for grounded robotics

Christopher Wang, Candace Ross, Yen-Ling Kuo, Boris Katz, Andrei Barbu

CoRL 2020

Children acquire their native language with apparent ease by observing how language is used in context and attempting to use it themselves. They do so without laborious annotations, negative examples, or even direct corrections. We take a step toward robots that can do the same by training a grounde

Cited by 0 Source ↗PDF ↗Scholar ↗

2019

ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models

Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund +2

NeurIPS 2019poster

We collect a large real-world test set, ObjectNet, for object recognition with controls where object backgrounds, rotations, and imaging viewpoints are random. Most scientific experiments have controls, confounds which are removed from the data, to ensure that subjects cannot perform a task by explo…

Cited by 697 Source ↗PDF ↗Scholar ↗

2018

Deep Sequential Models for Sampling-Based Planning

Yen-Ling Kuo, Andrei Barbu, Boris Katz

IROS 2018poster

We demonstrate how a sequence model and a sampling-based planner can influence each other to produce efficient plans and how such a model can automatically learn to take advantage of observations of the environment. Sampling-based planners such as RRT generally know nothing of their environments eve…

Cited by 30 Source ↗Scholar ↗