84 accepted papers
We present CineMPC, a complete cinematographic system that autonomously controls a drone to film multiple targets recording user-specified aesthetic objectives. Existing solutions in autonomous cinematography control only the camera extrinsics, namely, its position and orientation. In contrast, Cine
What makes a good viewpoint? The quality of the data used to learn 3D reconstructions is crucial for enabling efficient and accurate scene modeling. We study the active view selection problem and develop a principled analysis that yields a simple and interpretable criterion for selecting informative
Diffusion policies for visuomotor robot manipulation tasks achieve remarkable dexterity and robustness while only training on a small number of task demonstrations. However, the reason for this performance remains a mystery. In this paper, we offer a surprising hypothesis: diffusion policies essent…
It is currently challenging to deploy visuomotor robots at scale due to the potential of anomalous failures degrading performance, causing damage, or endangering human life. Bimanual manipulators are no exception; these robots have vast state spaces comprised of high-dimensional images and proprioce…
Autonomous drones capable of interpreting and executing high-level language instructions in unstructured environments remain a long-standing goal. Yet existing approaches are constrained by their dependence on hand-crafted skills, extensive parameter tuning, or computationally intensive models unsui
Learning robotic manipulation policies directly in the real world can be expensive and time-consuming. While reinforcement learning (RL) policies trained in simulation present a scalable alternative, effective sim-to-real transfer remains challenging, particularly for tasks that require precise dyna…
Large-scale robot learning has made progress on complex manipulation tasks, yet long-horizon, contact-rich problems—especially those involving deformable objects—remain challenging due to inconsistent demonstration quality. We propose a stage-aware, video-based reward modeling framework that jointly…
Large vision-language models have driven remarkable progress in open-vocabulary robot policies, e.g., generalist robot manipulation policies, that enable robots to complete complex tasks specified in natural language. Despite these successes, open-vocabulary autonomous drone navigation remains an un…
We propose SketchPlan, a diffusion-based planner that interprets 2D hand-drawn sketches over depth images to generate 3D flight paths for drone navigation. SketchPlan comprises two components: a SketchAdapter that learns to map the human sketches to projected 2D paths, and DiffPath, a diffusion mode
Mixture-of-Experts (MoE) has become a cornerstone in recent state-of-the-art large language models (LLMs). Traditionally, MoE relies on $\mathrm{Softmax}$ as the router score function to aggregate expert output, a designed choice that has persisted from the earliest MoE models to modern LLMs, and is…
We present VISTA (Viewpoint-based Image selection with Semantic Task Awareness), an active exploration method for robots to plan informative trajectories that improve 3D map quality in areas most relevant for task completion. Given an open-vocabulary search instruction (e.g., “find a person”), VISTA
We present VISTA (Viewpoint-based Image selection with Semantic Task Awareness), an active exploration method for robots to plan informative trajectories that improve 3D map quality in areas most relevant for task completion. Given an open-vocabulary search instruction (e.g., "find a person"), VISTA…
SAFER-Splat (Simultaneous Action Filtering and Environment Reconstruction) is a real-time, scalable, and minimally invasive safety filter, based on control barrier functions, for safe robotic navigation in a detailed map constructed at runtime using Gaussian Splatting (GSplat). We propose a novel Co
Generalizable long-horizon robotic assembly requires reasoning at multiple levels of abstraction. While end-to-end imitation learning (IL) is a promising approach, it typically requires large amounts of expert demonstration data and often struggles to achieve the high precision demanded by assembly…
Large language models (LLMs) have shown significant potential in guiding embodied agents to execute language instructions across a range of tasks, including robotic manipulation and navigation. However, existing methods are primarily designed for static environments and do not leverage the agent's o
Autonomous visual navigation is an essential element in robot autonomy. Reinforcement learning (RL) offers a promising policy training paradigm. However, existing RL methods suffer from high sample complexity, poor sim-to-real transfer, and limited runtime adaptability. These problems are particular
3D Gaussian Splatting offers expressive scene reconstruction and can model a broad range of visual, geometric, and semantic information. However, efficient real-time map reconstruction with data streamed from multiple robots and devices remains a challenge. To that end, we propose HAMMER, a server-b
We present Latent Theory of Mind (LatentToM), a decentralized diffusion policy architecture for collaborative robot manipulation. Our policy allows multiple manipulators with their own perception and computation to collaborate with each other towards a common task goal with or without explicit commu…
3D world models (i.e., learning-based 3D dynamics models) offer a promising approach to generalizable robotic manipulation by capturing the underlying physics of environment evolution conditioned on robot actions. However, existing 3D world models are primarily limited to single-material dynamics us…
The attention mechanism is a core component of the Transformer architecture. Various methods have been developed to compute attention scores, including multi-head attention (MHA), multi-query attention, group-query attention and so on. We further analyze the MHA and observe that its performance impr…
We present SIREN for registration of multi-robot Gaussian Splatting (GSplat) maps, with zero access to camera poses, images, and inter-map transforms for initialization or fusion of local submaps. To realize these capabilities, SIREN harnesses the versatility and robustness of semantics in three cri…
We propose a new simulator, training approach, and policy architecture, collectively called SOUS VIDE, for end-to-end visual drone navigation. Our trained policies exhibit zero-shot sim-to-real transfer with robust real-world performance using only onboard perception and computation. Our simulator,
This paper introduces CLIPSwarm, a new algorithm designed to automate the modeling of swarm drone formations based on natural language. The algorithm begins by enriching a provided word, to compose a text prompt that serves as input to an iterative approach to find the formation that best matches th…
This work explores conditions under which multi-finger grasping algorithms can attain robust sim-to-real transfer. While numerous large datasets facilitate learning *generative* models for multi-finger grasping at scale, reliable real-world dexterous grasping remains challenging, with most methods d…
With the rise of stochastic generative models in robot policy learning, end-to-end visuomotor policies are increasingly successful at solving complex tasks by learning from human demonstrations. Nevertheless, since real-world evaluation costs afford users only a small number of policy rollouts, it r
Multiple Unpiloted Aerial Vehicles (UAVs) working together have the potential to efficiently survey large geographical areas. Unfortunately, UAVs in the field may fail midway through a survey due to adverse weather, faster-than-expected battery drain, or mechanical malfunction, leaving part of the s
We present Splat-MOVER, a modular robotics stack for open-vocabulary robotic manipulation, which leverages the editability of Gaussian Splatting (GSplat) scene representations to enable multi-stage manipulation tasks. Splat-MOVER consists of: (i) ASK-Splat, a GSplat representation that distills sema…
Learning-based models for robot perception are known to suffer from two distinct sources of error: aleatoric and epistemic. Aleatoric uncertainty arises from inherently noisy training data and is easily quantified from residual errors during training. Conversely, epistemic uncertainty arises from a
In this work, we propose a novel method to supervise 3D Gaussian Splatting (3DGS) scenes using optical tactile sensors. Optical tactile sensors have become widespread in their use in robotics for manipulation and object representation; however, raw optical tactile sensor data is unsuitable to direct…
This work presents CineTransfer, an algorithmic framework that drives a robot to record a video sequence that mimics the cinematographic style of an input video. We propose features that abstract the aesthetic style of the input video, so the robot can transfer this style to a scene with visual deta…
Robotic applications often involve working in environments that are uncertain, dynamic, and partially observable. Recently, diffusion models have been proposed for learning trajectory prediction models trained from expert demonstrations, which can be used for planning in robot tasks. Such models hav…
We present a differentiable pipeline for simulating the motion of objects that represent their geometry as a continuous density field parameterized as a deep network. This includes Neural Radiance Fields (NeRFs), and other related models. From the density field, we estimate the dynamical properties
In active source seeking, a robot takes repeated measurements in order to locate a signal source in a cluttered and unknown environment. A key component of an active source seeking robot planner is a model that can produce estimates of the signal at unknown locations with uncertainty quantification.…
Game-theoretic motion planners are a potent solution for controlling systems of multiple highly interactive robots. Most existing game-theoretic planners unrealistically assume <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a priori</i> objective fu
Game-theoretic motion planners are a powerful tool for the control of interactive multi-agent robot systems. Indeed, contrary to predict-then-plan paradigms, game-theoretic planners do not ignore the interactive nature of the problem, and simultaneously predict the behaviour of other agents while co…
Neural Radiance Fields (NeRFs) have become a widely-applied scene representation technique in recent years, showing advantages for robot navigation and manipulation tasks. To further advance the utility of NeRFs for robotics, we propose a transformer-based framework, <monospace xmlns:mml="http://www
We present a differentiable formulation of rigid-body contact dynamics for objects and robots represented as compositions of convex primitives. Classical physics engines rely on non-differentiable collision detection modules. More recent optimization-based approaches simulating contact between conve
We present CineMPC, an algorithm to autonomously control a UAV-borne video camera in a nonlinear Model Predicted Control (MPC) loop. CineMPC controls both the position and orientation of the camera-the camera extrinsics-as well as the lens focal length, focal distance, and aperture-the camera intrin…
Many robotics problems, from robot motion planning to object manipulation, can be modeled as mixed-integer convex program (MICPs). However, state-of-the-art algorithms are still unable to solve MICPs for control problems quickly enough for online use and existing heuristics can typically only find s
We present DiNNO, a distributed algorithm that enables a group of robots to collaboratively optimize a deep neural network model while communicating over a mesh network. Each robot only has access to its own data and maintains its own version of the neural network, but eventually learns a model that
We present a method for autonomous exploration of large-scale unknown environments under mission time con-straints. We start by proposing the Frontloaded Information Gain Orienteering Problem (FIG-OP) - a generalization of the traditional orienteering problem where the assumption of a reliable envir…
We present a novel approach for risk-aware planning with human agents in multi-agent traffic scenarios. Our approach takes into account the wide range of human driver behaviors on the road, from aggressive maneuvers like speeding and overtaking, to conservative traits like driving slowly and conform…
Neural Radiance Fields (NeRFs) have recently emerged as a powerful paradigm for the representation of natural, complex 3D scenes. NeRFs represent continuous volumetric density and RGB values in a neural network, and generate photo-realistic images from unseen camera viewpoints through ray tracing. W
In this work we present a planning and control method for a quadrotor in an autonomous drone race. Our method combines the advantages of both model-based optimal control and model-free deep reinforcement learning. We consider a single drone racing on a track marked by a series of gates; through whic…
Existing game-theoretic planning methods assume that the robot knows the objective functions of the other agents a priori while, in practical scenarios, this is rarely the case. This letter introduces LUCIDGames, an inverse optimal control algorithm that is able to estimate the other agents' objecti
Successful robotic operation stochastic environments relies on accurate characterization of the underlying probability distributions, yet this is often imperfect due to limited knowledge. This work presents a control algorithm that is capable of handling such distributional mismatches. Specifically,
We present a method for computing exact reachable sets for deep neural networks with rectified linear unit (ReLU) activation. Our method is well-suited for use in rigorous safety analysis of robotic perception and control systems with deep neural network components. Our algorithm can compute both fo…
This paper presents Reduced State Value Iteration (RSVI), an algorithm to compute policies for Markov Decision Processes (MDPs) that have natural checkpoints, allowing for a solution based on a reduced state space. The algorithm is applied to find policies for multiple drones to persistently surveil…
Dexterous manipulation tasks often require contact switching, where fingers make and break contact with the object. We propose a method that plans trajectories for dexterous manipulation tasks involving contact switching using contact-implicit trajectory optimization (CITO) augmented with a high-lev…
Dynamic games are an effective paradigm for dealing with the control of multiple interacting actors. This paper introduces ALGAMES (Augmented Lagrangian GAME-theoretic Solver), a solver that handles trajectory optimization problems with multiple actors and general nonlinear state and input constrain…
Unmanned Aerial Vehicles (UAVs) are becoming increasingly popular in the film and entertainment industries, in part because of their maneuverability and perspectives they enable. While there exists methods for controlling the position and orientation of the drones for visibility, other artistic elem…
We propose a scalable cooperative control approach which coordinates a group of rigidly connected autonomous surface vessels to track desired trajectories in a planar water environment as a single floating modular structure. Our approach leverages the implicit information of the structure’s motion f…
We present a scalable distributed target tracking algorithm based on the alternating direction method of multipliers that is well-suited for a fleet of autonomous cars communicating over a vehicle-to-vehicle network. Each sensing vehicle communicates with its neighbors to execute iterations of a Kal…
In this paper, we consider a two-player racing game, where an autonomous ego vehicle has to be controlled to race against an opponent vehicle, which is either autonomous or human-driven. The approach to control the ego vehicle is based on a Sensitivity-ENhanced NAsh equilibrium seeking (SENNA) metho…
Modeling the stochastic behavior of interacting agents is key for safe motion planning. In this paper, we study the interaction of risk-aware agents in a game-theoretical framework. Under the entropic risk measure, we derive an iterative algorithm for approximating the intractable feedback Nash equi…
We study the problem of sequential task assignment and collision-free routing for large teams of robots in applications with inter-task precedence constraints (e.g., task A and task B must both be completed before task C may begin). Such problems commonly occur in assembly planning for robotic manuf…
This paper presents a novel online framework for safe crowd-robot interaction based on risk-sensitive stochastic optimal control, wherein the risk is modeled by the entropic risk measure. The sampling-based model predictive control relies on mode insertion gradient optimization for this risk measure…
We present a distributed algorithm to enable a group of robots to collaboratively manipulate an object to a desired configuration while avoiding obstacles. Each robot solves a local optimization problem iteratively and communicates with its local neighbors, ultimately converging to the optimal traje…
Most approaches to multi-robot control either rely on local decentralized control policies that scale well in the number of agents, or on centralized methods that can handle constraints and produce rich system-level behavior, but are typically computationally expensive and scale poorly in the number
In this work, we develop a novel decentralized coordination algorithm for a team of autonomous unmanned aerial vehicles (UAVs) to surveil an aggressive forest wildfire. For dangerous environmental processes that occur over very large areas, like forest wildfires, multi-agent systems cannot rely on l
We propose a nonlinear receding horizon game-theoretic planner for autonomous cars in competitive scenarios with other cars. The online planner is specifically formulated for a two car autonomous racing scenario in which each car tries to advance along a given track as far as possible with respect t…
Multi-robot assembly has the potential to greatly reduce the cost and risk associated with the fabrication of large structures. Using teams of robots to perform assembly offers numerous advantages such as parallelism, robustness to single-agent failures, and flexibility in scheduling and task assign…
This paper presents a novel distributed mapping algorithm for multiple resource-constrained robots operating in a rectilinear 2D environment. The algorithm is built upon the sparse wireframe map representation and updating framework in [1]. We propose an exploration strategy based on the labeling of…
To be successful in multi-player drone racing, a player must not only follow the race track in an optimal way, but also compete with other drones through strategic blocking, faking, and opportunistic passing while avoiding collisions. Since unveiling one's own strategy to the adversaries is not desi…
In this paper, we consider motion as a means of sending messages between robots. We focus on a scenario in which a message is encoded in a sending robot's trajectory, and decoded by a receiver robot equipped with a monocular camera. The relative pose between the robots is unknown. We introduce an on…
We present a framework to enable a fleet of rigidly attached quadrotor aerial robots to transport heavy objects along a known reference trajectory without inter-robot communication or centralized coordination. Leveraging a distributed wrench controller, we provide exponential stability guarantees fo…
This paper presents a design for a decentralized adaptive controller that allows a team of agents to manipulate a common payload in \mathbb{R}^{2}R2\mathbb{R}^{2} or \mathbb{R}^{3}R3\mathbb{R}^{3}. The controller requires no communication between agents and requires no a priori knowledge of agent po…
This paper proposes a distributed deep reinforcement learning (RL) based strategy for a team of Unmanned Aerial Vehicles (UAVs) to autonomously fight forest fires. We first model the forest fire as a Markov decision process (MDP) with a factored structure. We consider optimally controlling the fores…
This paper introduces the Buffered Input Cell as a reciprocal collision avoidance method for multiple vehicles with high-order linear dynamics, extending recently proposed methods based on the Buffered Voronoi Cell [1] and generalized Voronoi diagrams [2]. We prove that if each vehicle's control inp…
This paper presents a novel wireframe map structure for resource-constrained robots operating in a rectilinear 2D environment. The wireframe representation compactly represents geometry, in addition to transient situations such as occlusions and boundaries of unexplored regions. We formulate a parti…
This paper presents a novel wireframe map structure for resource-constrained robots operating in a rectilinear 2D environment. The wireframe representation compactly represents geometry, in addition to transient situations such as occlusions and boundaries of unexplored regions. We formulate a parti
This paper presents a novel multi-robot mapping algorithm which allows a large number of simple robots to map the discrete graphical structure underlying an environment of multiple disjoint subregions. Examples of such environments include rooms in a building, buildings in a town, chambers in a cave…
This letter presents a distributed collision avoidance algorithm for multiple dynamic vehicles moving in arbitrary dimensions. In our algorithm, each robot continually computes its buffered Voronoi cell (BVC) and plans its path within the BVC in a receding horizon fashion. We prove that our algorith
We propose a distributed algorithm for the cooperative pursuit of multiple evaders using multiple pursuers in a bounded convex environment. The algorithm is suitable for intercepting rogue drones in protected airspace, among other applications. The pursuers do not know the evaders' policy, but by us
We consider a class of robotic systems composed of high elongation linear actuators connected at universal joints. We derive the differential kinematics of such robots, and formalize concepts of controllability based on graph rigidity. Control methods are then developed for two separate applications…
Magnetic anomaly detection (MAD) is an important problem in applications ranging from geological surveillance to military reconnaissance. MAD sensors detect local disturbances in the magnetic field, which can be used to detect the existence of and to estimate the position of buried, hidden, or subme
This paper presents a method for controlling quadrotor swarms in an environment with obstacles, with an intuitive human-swarm interface operated by a single human user. Our method allows for the quadrotor swarm to maintain a desired formation, while also keeping the quadrotors a safe distance from o
We investigate the cooperative pursuit of an evader by a group of quadrotors in an environment with no-fly zones. While the pursuers cannot enter into no-fly zones, the evader may freely move through zones to avoid capture. Once the evader enters a no-fly zone, the pursuers calculate a reachable set…
In this paper we consider the problem of controlling a team of non-holonomic robots to reach a desired formation. The formation is described in terms of the desired relative positions and orientations the robots need to keep with respect to each other, and it is assumed that the robots do not have a
This paper presents a distributed method for navigating a team of robots in formation in 2D and 3D environments with static and dynamic obstacles. The robots are assumed to have a reduced communication and visibility radius and share information with their neighbors. Via distributed consensus the ro
This paper proposes a novel decentralized algorithm that coordinates the forces of a group of robots during a cooperative manipulation task. The highlight of our approach is that no communication is needed between any two robots. Our underlying intuition is that every follower robot can measure the
This paper proposes a new approach for a group of robots carrying out a collaborative task to adapt on-line to actuation performance variations among the robots. We consider the problem of multi-robot coverage, where a group of robots has to spread out to cover the environment. We suppose that some…
This paper presents a new control strategy to control a group of dog-like robots to drive a herd of non-cooperative sheep-like agents to a goal region in the environment. The sheep-like agents, which may be biological or robotic, respond to the presence of the dog-like robots with a repelling potent…
This paper proposes a method for controlling a team of quadrotor micro aerial vehicles to perform agile maneuvers while holding a fixed relative formation, as well as transitioning between a sequence of formations. The objective is to coordinate the quadrotors to fly in intricate interlaced patterns…