89 accepted papers
We propose CLEVER, an active learning system for robust semantic perception with Deep Neural Networks (DNNs). For data arriving in streams, our system seeks human support when encountering failures and adapts DNNs online based on human instructions. In this way, CLEVER can eventually accomplish the …
The Schrödinger Bridge provides a principled framework for modeling stochastic processes between distributions; however, existing methods are limited by energy-conservation assumptions, which constrains the bridge's shape preventing it from model varying-energy phenomena. To overcome this, we introd…
The ability to interpret and reason about spatial relations is fundamental for robotic manipulation tasks. For instance, a robot must understand that "inside" requires different geometric constraints than "touching", and "closer" involves dynamic changes in distance relationships. Despite progress i…
We propose MoRe-ERL, a framework that combines Episodic Reinforcement Learning (ERL) and residual learning, which refines preplanned reference trajectories into safe, feasible, and efficient task-specific trajectories. This framework is general enough to incorporate into arbitrary ERL methods and mo…
Human-like motion generation for robots often draws inspiration from biomechanical studies, which categorize complex human motions into hierarchical taxonomies. While these taxonomies provide rich structural information about how movements relate to one another, this information is frequently overlo…
By incorporating physical consistency as inductive bias, deep neural networks display increased generalization capabilities and data efficiency in learning nonlinear dynamic models. However, the complexity of these models generally increases with the system dimensionality, requiring larger datasets,…
We propose CLEVER, an active learning system for robust semantic perception with Deep Neural Networks (DNNs). For data arriving in streams, our system seeks human support when encountering failures and adapts DNNs online based on human instructions. In this way, CLEVER can eventually accomplish the
The online adaptation of exoskeleton control based on muscle activity sensing offers a promising approach to personalizing exoskeleton behavior based on the user's biosignals. While electromyography (EMG)-based methods have demonstrated improvements in joint torque estimation, EMG sensors require di
Accurately modeling and predicting complex dynamical systems, particularly those involving force exchange and dissipation, is crucial for applications ranging from fluid dynamics to robotics, but presents significant challenges due to the intricate interplay of geometric constraints and energy trans…
Bimanual manipulation is challenging due to precise spatial and temporal coordination required between two arms. While there exist several real-world bimanual systems, there is a lack of simulated benchmarks with a large task diversity for systematically studying bimanual capabilities across a wide
Humanoid robotic hands need to be versatile and capable of providing environmental information in order to serve as a platform for intelligent grasp control. To facilitate the design process of such hands, we present the KIT Robotic Hands. They have been designed to meet diverse application requirem
Ankle exoskeletons can assist the ankle joint and reduce the metabolic cost of walking. However, many existing ankle exoskeletons constrain the natural 3 degrees of freedom (DoF) of the ankle to limit the exoskeleton’s weight and mechanical complexity, thereby compromising comfort and kinematic comp…
Recent advances in task planning leverage Large Language Models (LLMs) to improve generalizability by combining such models with classical planning algorithms to address their inherent limitations in reasoning capabilities. However, these approaches face the challenge of dynamically capturing the in…
Advancements in Industry 4.0 demand sophisticated solutions for automatic robotic assembly sequence planning (RASP), capable of handling the diversity and complexity of modern manufacturing tasks. One approach to RASP is Assembly-by-Disassembly (AbD). It first searches for a disassembly sequence tha…
Visual imitation learning has achieved impressive progress in learning unimanual manipulation tasks from a small set of visual observations, thanks to the latest advances in computer vision. However, learning bimanual coordination strategies and complex object relations from bimanual visual demonstr…
Human motion taxonomies serve as high-level hierarchical abstractions that classify how humans move and interact with their environment. They have proven useful to analyse grasps, manipulation skills, and whole-body support poses. Despite substantial efforts devoted to design their hierarchy and und…
Artificial cognitive architectures traditionally rely on complex memory models to encode, store, and retrieve information. However, the conventional practice of transferring all data from working memory (WM) to long-term memory (LTM) leads to high data volumes and challenges in efficient information…
Executing bimanual manipulation tasks on humanoid robots introduces additional challenges due to inherent spatial and temporal coordination between both hands. In our previous work, we proposed the Bimanual Manipulation Taxonomy, which defines categories of bimanual manipulation strategies based on…
Movement primitives (MPs) are compact representations of robot skills that can be learned from demonstrations and combined into complex behaviors. However, merely equipping robots with a fixed set of innate MPs is insufficient to deploy them in dynamic and unpredictable environments. Instead, the fu…
Grasping with five-fingered humanoid hands is a complex control problem. Throughout the entire grasping motion, all finger joints need to be coordinated to achieve a stable grasp. Grasp synergies provide a simplified, low-dimensional representation of grasp postures and motions, that can be used for…
Learning task models of bimanual manipulation from human demonstration and their execution on a robot should take temporal constraints between actions into account. This includes constraints on (i) the symbolic level such as precedence relations or temporal overlap in the execution, and (ii) the sub…
To perform versatile mobile manipulation tasks in human-centered environments, the ability to efficiently transfer learned skills, knowledge, and experiences from one robot to another or across different environments is critical. In this paper, we present MAkEable, a versatile uni- and multi-manual…
We introduce Riemannian Flow Matching Policies (RFMP), a novel model for learning and synthesizing robot visuomotor policies. RFMP leverages the efficient training and inference capabilities of flow matching methods. By design, RFMP inherits the strengths of flow matching: the ability to encode high…
In this paper, we present an approach for learning collision-free robot trajectories in the presence of moving obstacles. As a first step, we train a backup policy to generate evasive movements from arbitrary initial robot states using model-free reinforcement learning. When learning policies for ot
With the rapid development of Large Language Models (LLMs), it is crucial to have benchmarks which can evaluate the ability of LLMs on different domains. One common use of LLMs is performing tasks on scientific topics, such as writing algorithms, querying databases or giving mathematical proofs. Ins…
Generating human-like robot motions is pivotal for achieving smooth human-robot interactions. Such motions contribute to better predictions of robot motions by humans, thus leading to more intuitive interaction and increased acceptability. Human likeness in robot motions has been conventionally meas…
In the realm of robotics, numerous downstream robotics tasks leverage machine learning methods for processing, modeling, or synthesizing data. Often, this data comprises variables that inherently carry geometric constraints, such as the unit-norm condition of quaternions representing rigid-body orie…
Task-oriented object grasping and rearrangement are key skills for robots, which have to perform versatile real-world manipulation tasks. However, they remain challenging due to partial observations of the objects and shape variations in categorical objects. In this paper, we present the Multi-featu…
Humans naturally execute many everyday manipulation actions with both arms simultaneously. Similarly, endowing robots with bimanual manipulation task models is key to efficiently perform complex manipulation tasks. To do so, a promising approach is to learn a library of task models from human demons…
In this paper, we present a method to engage measurement uncertainties with the probabilistic robustness to one system uncertainty measure. Providing a metric indicating the potential occurrence of dangerous situations is highly essential for safety-critical robot applications. Due to the difficulty…
The generation of energy-efficient and dynamic-aware robot motions that satisfy constraints such as joint limits, self-collisions, and collisions with the environment remains a challenge. In this context, Riemannian geometry offers promising solutions by identifying robot motions with geodesics on t…
Industry 4.0 facilitates a high number of product variants, posing significant challenges for modern manufacturing. One of them is the automatic creation of assembly sequences. This can be achieved with the assembly-by-disassembly (AbD) approach, which is currently highly inefficient. We aim at spee…
Obtaining reliable detections of a human is crucial for many safety-related robotic tasks. This can be done by human pose estimation methods, which predict the position of several different keypoints of the human body. In most cases, recent approaches based on neural networks produce ‘good’ results,…
The ability of humans to bimanually manipulate objects is unprecedented and has not been matched by robots yet. The goal-oriented coordination requires the consideration of both temporal and spatial constraints between the hands. Within this work, we propose a taxonomy which differentiates between t
Building humanoid robots with properties similar to those of humans in terms of strength and agility is a great and unsolved challenge. This work introduces a compact and lightweight wrist joint mechanism that is singularity-free and has large range of motion. The mechanism provides two degrees of f…
Dynamic motions of humans and robots are widely driven by posture-dependent nonlinear interactions between their degrees of freedom. However, these dynamical effects remain mostly overlooked when studying the mechanisms of human movement generation. Inspired by recent works, we hypothesize that huma…
Mobile manipulation tasks require a seamless integration of navigation and manipulation capabilities. Finding suitable robot placements to pick up and place objects in such tasks is crucial for time-efficient task execution. Sub-optimal robot placements result in infeasible solutions or require larg
The ability to detect failure during task execution and to recover from failure is vital for autonomous robots performing tasks in previously unknown environments. In this paper, we present an approach for failure detection during the execution of grasping and mobile manipulation tasks by a humanoid…
Learning temporal relations between actions in a bimanual manipulation task is important for capturing the constraints of actions required to achieve the task's goal. However, given several demonstrations of a bimanual manipulation task, the problem of identifying the true temporal dependencies betw…
In contrast to humans and animals who naturally execute seamless motions, learning and smoothly executing sequences of actions remains a challenge in robotics. This letter introduces a novel skill-agnostic framework that learns to sequence and blend skills based on differentiable optimization. Our a
For a robot to perform a grasping and manipulation task, it has to determine possible robot placements in the workspace, from which target objects or environmental elements relevant to the given task are reachable. This work presents a novel approach for finding placements for the mobile base of a h…
Robust vision-based grasping and manipulation of unknown objects in unstructured scenes requires the extraction of action candidates based on visual information while taking into account noise and occlusions in such scenes. We address this problem by combining the concept of affordances and <i>Bayes
Folding garments reliably and efficiently is a long standing challenge in robotic manipulation due to the complex dynamics and high dimensional configuration space of garments. An intuitive approach is to initially manipulate the garment to a canonical smooth configuration before folding. In this wo…
Intelligent, semi-autonomous prostheses take ad-vantage of combining autonomous functions and traditional myoelectric control. With the help of visual and environment sensors, intelligent prostheses achieve a level of autonomy which relieves the user from generating elaborate electromyographic (EMG)…
The ability to verbalize robot experience in natural language is key for a symbiotic human-robot interaction. While first works approached this problem using template-based verbalization on symbolic episode data only, we explore a novel way in which deep learning methods are used for the creation of
Engineering humanoid robot hands with the ability to dexterously grasp objects of different sizes, shapes, mate-rial properties and weights requires sophisticated tactile sensing and intelligent controllers able to interpret sensory information and adapt contact forces with the object to achieve a s…
We present a soft humanoid hand with in-finger integrated cameras and in-hand real-time image processing system for fast reactive grasping. Specifically, we describe an FPGA-based, in-hand integrated, embedded system for processing visual data captured by the five in-finger cameras while avoiding hi…
Bayesian optimization is a data-efficient technique which can be used for control parameter tuning, parametric policy adaptation, and structure design in robotics. Many of these problems require optimization of functions defined on non-Euclidean domains like spheres, rotation groups, or spaces of po…
Capturing scene dynamics and predicting the future scene state is challenging but essential for robotic manipulation tasks, especially when the scene contains both rigid and deformable objects. In this work, we contribute a simulation environment and generate a novel dataset for task-specific manipu…
We introduce a multi-functional robotic gripper equipped with a set of actions required for disassembly of electromechanical devices. The gripper consists of a robot arm with 5 degrees of freedom (DoF) for manipulation and a jaw gripper with a 1-DoF rotation joint and a 1-DoF closing joint. The syst…
This letter presents contact-safe Model-based Reinforcement Learning (MBRL) for robot applications that achieves contact-safe behaviors in the learning process. In typical MBRL, we cannot expect the data-driven model to generate accurate and reliable policies to the intended robotic tasks during the
Increasing the robustness of grasping actions and the recovery from failure is key to improving a robot’s autonomy. Endowing robots with the ability to robustly grasp and manipulate unknown difficult objects such as stones is required for sample collection in unknown environments. In this paper, we…
We present a novel underactued humanoid five finger soft hand, the KIT Finger-Vision Soft Hand, which is equipped with cameras in the fingertips and integrates a high performance embedded system for visual processing and control. We describe the actuation mechanism of the hand and the tendon-driven…
In real world applications, robotic solutions remain impractical due to the challenges that arise in unknown and unstructured environments. To perform complex manipulation tasks in complex and cluttered situations, robots need to be able to identify the interaction possibilities with the scene, i.e.…
Recognizing human actions is a vital task for a humanoid robot, especially in domains like programming by demonstration. Previous approaches on action recognition primarily focused on the overall prevalent action being executed, but we argue that bimanual human motion cannot always be described suff
Understanding the effects of actions is essential for planning and executing robot tasks. By imagining possible action consequences, a robot can choose specific action parameters to achieve desired goal states. We present an approach for parametrizing pushing actions based on learning internal predi…
Understanding spatial relations is a key element for natural human-robot interaction. Especially, a robot must be able to manipulate a given scene according to a human verbal command specifying desired spatial relations between objects. To endow robots with this ability, a suitable representation of…
Movement Primitives (MPs) are a promising way for representing robot motions in a flexible and adaptable manner. Due to the simple and compact form, they have been widely used in robotics. A major goal of the research activities on MPs is to learn models, which can adapt to changing task constraints…
Exoskeletons are considered to be a promising technology for assisting and augmenting human performance. A number of challenges related to design, intuitive control and interfaces to the human body must be addressed. In this paper, we approach the question of a minimal sensor setup for the realizati…
Tactile sensors have been increasingly used to support rigid robot grippers in object grasping and manipulation. However, rigid grippers are often limited in their ability to handle compliant, delicate, or irregularly shaped objects. In recent years, grippers made from soft and flexible materials ha…
Credit assignment in Meta-reinforcement learning (Meta-RL) is still poorly understood. Existing methods either neglect credit assignment to pre-adaptation behavior or implement it naively. This leads to poor sample-efficiency during meta-training as well as ineffective task identification strategies…
Despite impressive advances of humanoid robotics, the autonomous planning of whole-body loco-manipulation actions in unknown environments is still an open problem. In our previous work, we addressed two fundamental aspects related to this problem: 1) the autonomous detection of end-effector contact…
We propose a computational formalization of affordances, which is able to consistently combine affordance-related evidence resulting from different observations. We represent affordances as Dempster-Shafer belief functions defined over the space of end-effector poses, which can be combined using unc
Dynamic systems are a practical alternative to motion planning in executing robot actions. They are of particular interest in Learning from Demonstration, as here we aim to carry out actions in a certain fashion, without a model or in-depth knowledge about the world, which might be difficult to achi…
We present a novel deep neural network architecture for representing robot experiences in an episodic-like memory that facilitates encoding, recalling, and predicting action experiences. Our proposed unsupervised deep episodic memory model as follows: First, encodes observed actions in a latent vect
The paper presents and evaluates a distance-aware dynamic roadmap (DA-DRM) algorithm as an extension of the dynamic roadmap (DRM) approach. In contrast to previous work, the algorithm is capable of planning collision-free trajectories while considering the distance to obstacles, even in unknown envi
Tactile sensing of surface normals is essential for exploration of unknown objects. Many tactile sensors have been developed for contact measurement. However, few of these sensors provide surface orientation, and only up to a limited degree. This paper presents a novel contact and surface orientatio…
Reliable execution of robot manipulation actions in cluttered environments requires that the robot is able to understand relations between objects and reason about consequences of actions applied to these objects. We present an approach for extracting physically plausible support relations between o
We present a data-driven, bottom-up, deep learning approach to robotic grasping of unknown objects using Deep Convolutional Neural Networks (DCNNs). The approach uses depth images of the scene as its sole input for synthesis of a single-grasp solution during execution, adequately portraying the robo…
Intuitive exoskeleton control is fundamental since it contributes to improved user acceptance and wearability comfort. This requires the detection of user's motion intention and its incorporation into the exoskeleton control system. In this work, we propose a classification system based on Hidden Ma…
Model-based reinforcement learning approaches carry the promise of being data efficient. However, due to challenges in learning dynamics models that sufficiently match the real-world dynamics, they struggle to achieve the same asymptotic performance as model-free methods. We propose Model-Based Meta
Deep reinforcement learning (RL) methods generally engage in exploratory behavior through noise injection in the action space. An alternative is to add noise directly to the agent's parameters, which can lead to more consistent exploration and a richer set of behaviors. Methods such as evolutionary…
In this letter, we present a grasp planner that integrates two sources of information to generate robust grasps for a robotic hand. First, the topological information of the object model is incorporated by building the mean curvature skeleton and segmenting the object accordingly in order to identif
The development and control of prosthetic hands is an active research area and recently progress in mechatronics, sensor integration and innovative control has been made. However, integration of different components into a prosthetic hand remains challenging due to space constraints, the requirement…
This work presents the concept of a robotic gripper designed for the disassembly of electromechanical devices that comprises several innovative ideas. Novel concepts include the ability to interchange built-in tools without the need to grasp them, the ability to reposition grasped objects in-hand, t…
Elderly or disabled people usually need augmented nursing attention both in home and clinical environments, especially to perform bathing activities. The development of an assistive robotic bath system, which constitutes a central motivation of this letter, would increase the independence and safety
Robotic coverage path planning describes the problem of determining a configuration space trajectory for successively covering a specified workspace target area with the robot's end-effector. Performing coverage path planning for mobile robots further requires solving the problem of robot placement,…
To increase the autonomy of humanoid robots, the visual perception must support the efficient collection and interpretation of visual scene cues by providing task-dependent information. Active vision systems allow to extend the observable workspace by employing active gaze control, i.e. by shifting…
An important question in imitation learning is how to generalize a learned motion to novel situations. The motion generalization depends on a set of features which can be represented as feature vectors spanning a feature space, called query space. The purpose of generalization is to find a mapping f…
We present a novel unsupervised framework, which links continuous visual features and symbolic textual descriptions of manipulation activity videos. First, we extract the semantic representation of visually observed manipulations by applying a bottom-up approach to the continuous image streams. We t
Dynamic movement primitives prove to be a useful and effective way to represent a movement of a given agent. However, the original DMP formulation does not take the interaction among multiple agents into the consideration. Thus, many researchers focus on the development of a coupling term for the un…
Object shape information is essential for robot manipulation tasks, in particular for grasp planning and collision-free motion planning. But in general a complete object model is not available, in particular when dealing with unknown objects. We propose a method for completing shapes that are only p…
We address the question of how resource-aware concepts can be utilized in motion planning algorithms. Resource-awareness facilitate better resource allocation on global system level, e.g. when a humanoid robot needs to distribute and schedule a wide variety of concurrent algorithms. We present a mot…
We propose a formalism for the hierarchical representation of affordances. Starting with a perceived model of the environment consisting of geometric primitives like planes or cylinders, we define a hierarchical system for affordance extraction whose foundation are elementary power grasp affordances…
We present a novel approach for generating sequences of whole-body poses with multi-contacts for humanoid robots, which is inspired by techniques from natural language processing. To this end, we propose a probabilistic n-gram language model learned from observation of human locomotion tasks. Human…
This paper presents the development of a new type of robot capable of vertical and directional jumping. The robot uses soft silicone elastomer based pneumatic actuators as legs that accelerate the platform upwards by rapid pressurization. The robot is able to control and adjust the direction of the…
Exploiting interaction with the environment is a promising and powerful way to enhance stability of humanoid robots and robustness while executing locomotion and manipulation tasks. Recently some works have started to show advances in this direction considering humanoid locomotion with multi-contact…
In this work we present a fast kinodynamic RRT-planner that uses dynamic nonprehensile actions to rearrange cluttered environments. In contrast to many previous works, the presented planner is not restricted to quasi-static interactions and monotonicity. Instead the results of dynamic robot actions…
We present a randomized kinodynamic planner that solves rearrangement planning problems. We embed a physics model into the planner to allow reasoning about interaction with objects in the environment. By carefully selecting this model, we are able to reduce our state and action space, gaining tracta…