AIConfPaper
← Search

Jonathan Kelly

38 accepted papers

2026

A Photorealistic Dataset and Vision-Based Algorithm for Anomaly Detection During Proximity Operations in Lunar Orbit

Selina Leveugle, Chang Won Lee, Svetlana Stolpner, Chris Langley, Paul Grouchy, Steven L. Waslander +1
RA-L 2026

NASA's forthcoming Lunar Gateway space station, which will be uncrewed most of the time, will need to operate with an unprecedented level of autonomy. One key challenge is enabling the Canadarm3, the Gateway's external robotic system, to detect hazards in its environment using its onboard inspection

Cited by 0Source ↗code ↗Scholar ↗
2026

A Photorealistic Dataset and Vision-Based Algorithm for Anomaly Detection During Proximity Operations in Lunar Orbit

Selina Leveugle, Chang Won Lee, Svetlana Stolpner, Chris Langley, Paul Grouchy, Steven Lake Waslander +1
ICRA 2026poster

NASA's forthcoming Lunar Gateway space station, which will be uncrewed most of the time, will need to operate with an unprecedented level of autonomy. One key challenge is enabling the Canadarm3, the Gateway's external robotic system, to detect hazards in its environment using its onboard inspection…

Cited by 0Source ↗PDF ↗Scholar ↗
2026

Generative Graphical Inverse Kinematics (Abstract Reprint)

Oliver Limoyo, Filip Marić, Matthew Giamou, Petra Alexson, Ivan Petrović, Jonathan Kelly
AAAI 2026technical

Quickly and reliably finding accurate inverse kinematics (IK) solutions remains a challenging problem for many robot manipulators. Existing numerical solvers are broadly applicable but typically only produce a single solution and rely on local search techniques to minimize nonconvex objective functi

Cited by 0Source ↗PDF ↗Scholar ↗
2025

Automated Planning Domain Inference for Task and Motion Planning

Jinbang Huang, Allen Tao, Rozilyn Marco, Miroslav Bogdanovic, Jonathan Kelly, Florian Shkurti
ICRA 2025

Task and motion planning (TAMP) frameworks address long and complex planning problems by integrating high-level task planners with low-level motion planners. However, existing TAMP methods rely heavily on the manual design of planning domains that specify the preconditions and postconditions of all

Cited by 5Source ↗Scholar ↗
2025

Efficient Imitation Without Demonstrations via Value-Penalized Auxiliary Control from Examples

Trevor Ablett, Bryan Chan, Jayce Haoran Wang, Jonathan Kelly
ICRA 2025

Common approaches to providing feedback in reinforcement learning are the use of hand-crafted rewards or full-trajectory expert demonstrations. Alternatively, one can use examples of completed tasks, but such an approach can be extremely sample inefficient. We introduce value-penalized auxiliary con

Cited by 0Source ↗code ↗Scholar ↗
2024

PhotoBot: Reference-Guided Interactive Photography via Natural Language

Oliver Limoyo, Jimmy Li, Dmitriy Rivkin, Jonathan Kelly, Gregory Dudek
IROS 2024poster

We introduce PhotoBot, a framework for fully automated photo acquisition based on an interplay between high-level human language guidance and a robot photographer. We propose to communicate photography suggestions to the user via reference images that are selected from a curated gallery. We leverage…

Cited by 1Source ↗Scholar ↗
2024

Watch Your Steps: Local Image and Scene Editing by Text Instructions

Ashkan Mirzaei*, Tristan T Aumentado-Armstrong, Marcus A Brubaker, Jonathan Kelly, Alex Levinshtein, Konstantinos G Derpanis +1
ECCV 2024oral

"The success of denoising diffusion models in generating and editing images has sparked interest in using diffusion models for editing 3D scenes represented via neural radiance fields (NeRFs). However, current 3D editing methods lack a way to both pinpoint the edit location and limit changes to the…

Cited by 35Source ↗PDF ↗Scholar ↗
2024

Working Backwards: Learning to Place by Picking

Oliver Limoyo, Abhisek Konar, Trevor Ablett, Jonathan Kelly, Francois R. Hogan, Gregory Dudek
IROS 2024poster

We present placing via picking (PvP), a method to autonomously collect real-world demonstrations for a family of placing tasks in which objects must be manipulated to specific, contact-constrained locations. With PvP, we approach the collection of robotic object placement demonstrations by reversing…

Cited by 0Source ↗Scholar ↗
2023

CIDGIKc: Distance-Geometric Inverse Kinematics for Continuum Robots

Hanna Jiamei Zhang, Matthew Giamou, Filip Maric, Jonathan Kelly, Jessica Burgner-Kahrs
RA-L 2023

The small size, high dexterity, and intrinsic compliance of continuum robots (CRs) make them well suited for constrained environments. Solving the inverse kinematics (IK), that is finding robot joint configurations that satisfy desired position or pose queries, is a fundamental challenge in motion p

Cited by 8Source ↗Scholar ↗
2023

Learning From Guided Play: Improving Exploration for Adversarial Imitation Learning With Simple Auxiliary Tasks

Trevor Ablett, Bryan Chan, Jonathan Kelly
RA-L 2023

Adversarial imitation learning (AIL) has become a popular alternative to supervised imitation learning that reduces the distribution shift suffered by the latter. However, AIL requires effective exploration during an online reinforcement learning phase. In this work, we show that the standard, naïve

Cited by 13Source ↗code ↗Scholar ↗
2023

Reference-guided Controllable Inpainting of Neural Radiance Fields

Ashkan Mirzaei, Tristan Aumentado-Armstrong, Marcus A. Brubaker, Jonathan Kelly, Alex Levinshtein, Konstantinos G. Derpanis +1
ICCV 2023poster

The popularity of Neural Radiance Fields (NeRFs) for view synthesis has led to a desire for NeRF editing tools. Here, we focus on inpainting regions in a view-consistent and controllable manner. In addition to the typical NeRF inputs and masks delineating the unwanted region in each view, we require…

Cited by 42PDF ↗code ↗Scholar ↗
2023

SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting With Neural Radiance Fields

Ashkan Mirzaei, Tristan Aumentado-Armstrong, Konstantinos G. Derpanis, Jonathan Kelly, Marcus A. Brubaker, Igor Gilitschenski +1
CVPR 2023poster

Neural Radiance Fields (NeRFs) have emerged as a popular approach for novel view synthesis. While NeRFs are quickly being adapted for a wider set of applications, intuitively editing NeRF scenes is still an open challenge. One important editing task is the removal of unwanted objects from a 3D scene…

Cited by 140Source ↗PDF ↗code ↗Scholar ↗
2023

The Sum of Its Parts: Visual Part Segmentation for Inertial Parameter Identification of Manipulated Objects

Philippe Nadeau, Matthew Giamou, Jonathan Kelly
ICRA 2023poster

To operate safely and efficiently alongside human workers, collaborative robots (cobots) require the ability to quickly understand the dynamics of manipulated objects. However, traditional methods for estimating the full set of inertial parameters rely on motions that are necessarily fast and unsafe…

Cited by 6Source ↗code ↗Scholar ↗
2022

Convex Iteration for Distance-Geometric Inverse Kinematics

Matthew Giamou, Filip Maric, David M. Rosen, Valentin Peretroukhin, Nicholas Roy, Ivan Petrovic +1
RA-L 2022

Inverse kinematics (IK) is the problem of finding robot joint configurations that satisfy constraints on the position or pose of one or more end-effectors. For robots with redundant degrees of freedom, there is often an infinite, nonconvex set of solutions. The IK problem is further complicated when

Cited by 31Source ↗code ↗Scholar ↗
2022

Fast Object Inertial Parameter Identification for Collaborative Robots

Philippe Nadeau, Matthew Giamou, Jonathan Kelly
ICRA 2022poster

Collaborative robots (cobots) are machines designed to work safely alongside people in human-centric environments. Providing cobots with the ability to quickly infer the inertial parameters of manipulated objects will improve their flexibility and enable greater usage in manufacturing and other area…

Cited by 12Source ↗code ↗Scholar ↗
2022

LaTeRF: Label and Text Driven Object Radiance Fields

Ashkan Mirzaei, Yash Kant, Jonathan Kelly, Igor Gilitschenski
ECCV 2022poster

"Obtaining 3D object representations is important for creating photo-realistic simulators and collecting assets for AR/VR applications. Neural fields have shown their effectiveness in learning a continuous volumetric representation of a scene from 2D images, but acquiring object representations from…

Cited by 37Source ↗PDF ↗Scholar ↗
2022

Learning to Detect Slip with Barometric Tactile Sensors and a Temporal Convolutional Neural Network

Abhinav Grover, Philippe Nadeau, Christopher Grebe, Jonathan Kelly
ICRA 2022poster

The ability to perceive object slip via tactile feedback enables humans to accomplish complex manipulation tasks including maintaining a stable grasp. Despite the utility of tactile information for many applications, tactile sensors have yet to be widely deployed in industrial robotics settings; par…

Cited by 13Source ↗Scholar ↗
2022

On the Coupling of Depth and Egomotion Networks for Self-Supervised Structure from Motion

Brandon Wagstaff, Valentin Peretroukhin, Jonathan Kelly
RA-L 2022

Structure from motion (SfM) has recently been formulated as a self-supervised learning problem, where neural network models of depth and egomotion are learned jointly through view synthesis. Herein, we address the open problem of how to best <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xml

Cited by 9Source ↗code ↗Scholar ↗
2021

A Continuous-Time Approach for 3D Radar-to-Camera Extrinsic Calibration

Emmett Wise, Juraj Peršić, Christopher Grebe, Ivan Petrović, Jonathan Kelly
ICRA 2021poster

Reliable operation in inclement weather is essential to the deployment of safe autonomous vehicles (AVs). Robustness and reliability can be achieved by fusing data from the standard AV sensor suite (i.e., lidars, cameras) with weather robust sensors, such as millimetre-wavelength radar. Critically,…

Cited by 38Source ↗Scholar ↗
2021

Learned Camera Gain and Exposure Control for Improved Visual Feature Detection and Matching

Justin Tomasi, Brandon Wagstaff, Steven L. Waslander, Jonathan Kelly
RA-L 2021

Successful visual navigation depends upon capturing images that contain sufficient useful information. In this letter, we explore a data-driven approach to account for environmental lighting changes, improving the quality of images for use in visual odometry (VO) or visual simultaneous localization

Cited by 38Source ↗Scholar ↗
2021

Seeing All the Angles: Learning Multiview Manipulation Policies for Contact-Rich Tasks from Demonstrations

Trevor Ablett, Yifan Zhai, Jonathan Kelly
IROS 2021poster

Learned visuomotor policies have shown considerable success as an alternative to traditional, hand-crafted frameworks for robotic manipulation. Surprisingly, an extension of these methods to the multiview domain is relatively unexplored. A successful multiview policy could be deployed on a mobile ma…

Cited by 4Source ↗code ↗Scholar ↗
2020

A Smooth Representation of Belief over SO(3) for Deep Rotation Learning with Uncertainty

Valentin Peretroukhin, Matthew Giamou, W. Nicholas Greene, David Rosen, Jonathan Kelly, Nicholas Roy
RSS 2020poster

Accurate rotation estimation is at the heart of robot perception tasks such as visual odometry and object pose estimation. Deep neural networks have provided a new way to perform these tasks, and the choice of rotation representation is an important part of network design. In this work, we present a…

Cited by 85Source ↗PDF ↗code ↗Scholar ↗
2020

Heteroscedastic Uncertainty for Robust Generative Latent Dynamics

Oliver Limoyo, Bryan Chan, Filip Maric, Brandon Wagstaff, A. Rupam Mahmood, Jonathan Kelly
RA-L 2020

Learning or identifying dynamics from a sequence of high-dimensional observations is a difficult challenge in many domains, including reinforcement learning, and control. The problem has recently been studied from a generative perspective through latent dynamics: high-dimensional observations are em

Cited by 9Source ↗code ↗Scholar ↗
2020

Inverse Kinematics for Serial Kinematic Chains via Sum of Squares Optimization

Filip Marić, Matthew Giamou, Soroush Khoubyarian, Ivan Petrović, Jonathan Kelly
ICRA 2020poster

Inverse kinematics is a fundamental challenge for articulated robots: fast and accurate algorithms are needed for translating task-related workspace constraints and goals into feasible joint configurations. In general, inverse kinematics for serial kinematic chains is a difficult nonlinear problem,…

Cited by 23Source ↗code ↗Scholar ↗
2020

Learning Matchable Image Transformations for Long-Term Metric Visual Localization

Lee E. Clement, Mona Gridseth, Justin Tomasi, Jonathan Kelly
RA-L 2020

Long-term metric self-localization is an essential capability of autonomous mobile robots, but remains challenging for vision-based systems due to appearance changes caused by lighting, weather, or seasonal variations. While experience-based mapping has proven to be an effective technique for bridgi

Cited by 16Source ↗code ↗Scholar ↗
2020

Self-Supervised Deep Pose Corrections for Robust Visual Odometry

Brandon Wagstaff, Valentin Peretroukhin, Jonathan Kelly
ICRA 2020poster

We present a self-supervised deep pose correction (DPC) network that applies pose corrections to a visual odometry estimator to improve its accuracy. Instead of regressing inter-frame pose changes directly, we build on prior work that uses data-driven learning to regress pose corrections that accoun…

Cited by 30Source ↗code ↗Scholar ↗
2019

Certifiably Globally Optimal Extrinsic Calibration From Per-Sensor Egomotion

Matthew Giamou, Ziye Ma, Valentin Peretroukhin, Jonathan Kelly
RA-L 2019

We present a certifiably globally optimal algorithm for determining the extrinsic calibration between two sensors that are capable of producing independent egomotion estimates. This problem has been previously solved using a variety of techniques, including local optimization approaches that have no

Cited by 31Source ↗code ↗Scholar ↗
2019

Fast Manipulability Maximization Using Continuous-Time Trajectory optimization

Filip Marić, Oliver Limoyo, Luka Petrović, Trevor Ablett, Ivan Petrović, Jonathan Kelly
IROS 2019poster

A significant challenge in manipulation motion planning is to ensure agility in the face of unpredictable changes during task execution. This requires the identification and possible modification of suitable joint-space trajectories, since the joint velocities required to achieve a specific endeffec…

Cited by 21Source ↗Scholar ↗
2019

The Phoenix Drone: An Open-Source Dual-Rotor Tail-Sitter Platform for Research and Education

Yilun Wu, Xintong Du, Rikky Duivenvoorden, Jonathan Kelly
ICRA 2019poster

In this paper, we introduce the Phoenix drone: the first completely open-source tail-sitter micro aerial vehicle (MAV) platform. The vehicle has a highly versatile, dual-rotor design and is engineered to be low-cost and easily extensible/modifiable. Our open-source release includes all of the design…

Cited by 12Source ↗code ↗Scholar ↗
2018

How to Train a CAT: Learning Canonical Appearance Transformations for Direct Visual Localization Under Illumination Change

Lee E. Clement, Jonathan Kelly
RA-L 2018

Direct visual localization has recently enjoyed a resurgence in popularity with the increasing availability of cheap mobile computing power. The competitive accuracy and robustness of these algorithms compared to state-of-the-art feature-based methods, as well as their natural ability to yield dense

Cited by 26Source ↗code ↗Scholar ↗
2018

Near-Optimal Budgeted Data Exchange for Distributed Loop Closure Detection

Yulun Tian, Kasra Khosoussi, Matthew Giamou, Jonathan How, Jonathan Kelly
RSS 2018poster

Inter-robot loop closure detection is a core problem in collaborative SLAM (CSLAM). Establishing inter-robot loop closures is a resource-demanding process, during which robots must consume a substantial amount of mission-critical resources (e.g., battery and bandwidth) to exchange sensory data. Howe…

Cited by 25Source ↗PDF ↗Scholar ↗
2018

Self-Calibration of Mobile Manipulator Kinematic and Sensor Extrinsic Parameters Through Contact-Based Interaction

Oliver Limoyo, Trevor Ablett, Filip Marić, Luke Volpatti, Jonathan Kelly
ICRA 2018poster

We present a novel approach for mobile manipulator self-calibration using contact information. Our method, based on point cloud registration, is applied to estimate the extrinsic transform between a fixed vision sensor mounted on a mobile base and an end effector. Beyond sensor calibration, we demon…

Cited by 17Source ↗Scholar ↗
2017

Reducing drift in visual odometry by inferring sun direction using a Bayesian Convolutional Neural Network

Valentin Peretroukhin, Lee Clement, Jonathan Kelly
ICRA 2017poster

We present a method to incorporate global orientation information from the sun into a visual odometry pipeline using only the existing image stream, where the sun is typically not visible. We leverage recent advances in Bayesian Convolutional Neural Networks to train and implement a sun detection mo…

Cited by 42Source ↗code ↗Scholar ↗
2016

PROBE-GK: Predictive robust estimation using generalized kernels

Valentin Peretroukhin, William Vega-Brown, Nicholas Roy, Jonathan Kelly
ICRA 2016

Many algorithms in computer vision and robotics make strong assumptions about uncertainty, and rely on the validity of these assumptions to produce accurate and consistent state estimates. In practice, dynamic environments may degrade sensor performance in predictable ways that cannot be captured wi

Cited by 19Source ↗Scholar ↗
2015

PROBE: Predictive robust estimation for visual-inertial navigation

Valentin Peretroukhin, Lee Clement, Matthew Giamou, Jonathan Kelly
IROS 2015poster

Navigation in unknown, chaotic environments continues to present a significant challenge for the robotics community. Lighting changes, self-similar textures, motion blur, and moving objects are all considerable stumbling blocks for state-of-the-art vision-based navigation algorithms. In this paper w…

Cited by 27Source ↗Scholar ↗