44 accepted papers
Universal jamming grippers excel at grasping unknown objects due to their compliant bodies. Traditional tactile sensors can compromise this compliance, reducing grasping performance. We present acoustic sensing as a form of morphological sensing, where the gripper's soft body itself becomes the sens…
Robotic affordance estimation is challenging due to visual, geometric, and semantic ambiguities in sensory input. We propose a method that disambiguates these signals using two coupled recursive estimators for sub-aspects of affordances: graspable and movable regions. Each estimator encodes property…
Visual uncertainties such as occlusions, lack of texture, and noise present significant challenges in obtaining accurate kinematic models for safe robotic manipulation. We introduce a probabilistic real-time approach that leverages the human hand as a prior to mitigate these uncertainties. By tracki
We introduce a novel gradient-based approach for solving sequential tasks by dynamically adjusting the underlying myopic potential field in response to feedback and the world's regularities. This adjustment implicitly considers subgoals encoded in these regularities, enabling the solution of long se
A robotic system’s hardware and control policy must be co-optimized to ensure they complement each other to interact robustly with the environment. However, this combined search is extremely high-dimensional and intractable without a suitable underlying representation. This paper uses environmental…
We introduce a Learning from Demonstration (LID) approach for contact-rich manipulation tasks, i.e., tasks in which the manipulandum's motion is constrained by contact with the environment. Our approach is motivated by the insight that even a large number of demonstrations will often not contain suf…
We present a robust method to visually segment scenes into objects based on motion and appearance. Both these cues provide complementary information that we fuse using two interconnected recursive estimators: One estimates object segmentation from motion as a probabilistic clustering of tracked 3D p…
This paper presents a feedback-control framework for in-hand manipulation (IHM) with dexterous soft hands that enables the acquisition of manipulation skills in the real-world within minutes. We choose the deformation state of the soft hand as the control variable. To control for a desired deformati…
Robots need to understand articulated objects, such as drawers. The state of articulated structures is commonly estimated using vision, but visual perception is limited when objects are occluded, have few salient features, or are not in the camera's field of view. Audio sensing does not face these c…
We present a simple approach to in-hand cube reconfiguration. By simplifying planning, control, and perception as much as possible, while maintaining robust and general performance, we gain insights into the inherent complexity of in-hand cube reconfiguration. We also demonstrate the effectiveness o…
Soft robotics is an emerging field that yields promising results for tasks that require safe and robust interactions with the environment or with humans, such as grasping, manipulation, and human-robot interaction. Soft robots rely on intrinsically compliant components and are difficult to equip wit…
We create a virtual 2D tactile array for soft pneumatic actuators using embedded audio components. We detect contact-specific changes in sound modulation to infer tactile information. We evaluate different sound representations and learning methods to detect even small contact variations. We demonst…
We present a novel learning from demonstration approach which uses environmental constraints as the underlying representation to interpret and reproduce demonstrations. This representation based on environmental constraints separates the information that facilitates generalization from the informati
A gaze-fixating robot perceives distance to the fixated object and relative positions of surrounding objects immediately, accurately, and robustly. We show how fixation, which is the act of looking at one object while moving, exploits regularities in the geometry of 3D space to obtain this informati…
Robotic manipulation behavior should be robust to disturbances that violate high-level task-structure. Such robustness can be achieved by constantly monitoring the environment to observe the discrete high-level state of the task. This is possible because different phases of a task are characterized…
This paper offers an explanation of why humans can effortlessly grasp objects from a pile. We identified a regularity in objects’ motion when pushed, namely, an object separates and stabilizes in front of the pusher. We devise an open-loop grasping strategy leveraging this regularity in piles of nea…
We present an active acoustic sensor that turns soft pneumatic actuators into contact sensors. The whole surface of the actuator becomes a sensor, rendering the question of where best to place a contact sensor unnecessary. At the same time, the compliance of the soft actuator remains unaffected. A s…
In this work, we investigate the behavior of artificial hands under impulsive load conditions. Resilience to impacts has been seldom considered in grasp and manipulation literature and benchmarks, although it is one of the most relevant issues in a number of applications involving physical interacti
The space of all possible deformations of soft robotic actuators is extremely large. It is impossible to explicitly measure each internal degree of freedom, regardless of the number and types of sensors. It is, however, possible to measure a smaller subset of task-relevant deformations using only a…
We introduce Recurrent State Representation Learning (RSRL) to tackle the problem of state representation learning in robotics for partially observable environments. To learn low-dimensional state representations, we combine a Long Short Term Memory network with robotic priors. RSRL introduces new p…
We propose a novel sensing method for soft pneumatic actuators. The method uses a single microphone, embedded into the actuator's air chamber. Contact with the environment induces sound (vibration) in the actuator. The materials and the shape of the actuator reflect, refract, and attenuate the sound…
A robot with contact sensing capability can reduce uncertainty relative to the environment by deliberately moving into contact and matching the resulting contact measurement to different possible states in the world. We present a manipulation planner that finds and sequences these actions by reasoni…
We demonstrate that moving the wrist while the fingers perform a grasp increases performance. The coordination shapes the interactions between the fingers, the object and its environment to extend the hand capabilities (e.g. higher payload and precision). We evaluated our hypothesis with a human gra…
We present differentiable particle filters (DPFs): a differentiable implementation of the particle filter algorithm with learnable motion and measurement models. Since DPFs are end-to-end differentiable, we can efficiently train their models by optimizing end-to-end state estimation performance, rat…
In the context of robotic manipulation and grasping, the shift from a view that is static (force closure of a single posture) and contact-deprived (only contact for force closure is allowed, everything else is obstacle) towards a view that is dynamic and contact-rich (soft manipulation) has led to a…
Interactive perception exploits the correlation between forceful interactions and changes in the observed signals to extract task-relevant information from the sensor stream. Finding the most informative interactions to perceive complex objects, like articulated mechanisms, is challenging because th…
The compliance of soft actuators makes manipulation safer and simplifies control. But their high flexibility also makes sensorization challenging. From the large space of possible deformations not all are equally important. We present a method for sensorization of soft actuators that, for a given ap…
The most prominent criterion for learning of manipulation skills is the optimization of task success, modeled as expected reward or probability of success. This is sensible if we only want to optimize a single controller. But if learned manipulation primitives are used as modules in a larger system,…
To leverage soft hands to their full potential for grasping, we propose to design their morphology and control signals together. Considering both parameter domains makes it easier and faster to find solutions compared to fixing parameters of either domain. Additionally, the approach scales well to h…
We present an online system to perceive kinematic properties of articulated objects from multi-modal sensor streams. The novelty of our system is that it leverages multi-modal information in a cross-modal manner: instead of simply fusing information from different modalities, sensor streams are inte…
Handshakes are common greetings, and humans therefore have strong priors of what a handshake should feel like. This makes it challenging to create compelling and realistic human-robot handshakes, necessitating the consideration of human haptic perception in the design of robot hands. At its most bas…
In this paper we present a planner that interleaves free-space motion with motion in contact to reduce uncertainty. The planner finds such motions by growing a search tree in the combined space of collision-free and contact configurations. The planner reasons efficiently about the accumulated uncert…
In many robotic applications, softness leads to improved performance, robustness, and safety, while lowering manufacturing cost, increasing versatility, and simplifying control. The advantages of soft robots derive from the fact that their behavior partially results from interactions of the robot's…
Environment-constrained grasping exploits beneficial interactions between hand, object, and environment to increase grasp success. Instead of focusing on the final static relationship between hand posture and object pose, this view of grasping emphasizes the need and the opportunity to select the mo…
Observations of human grasping reveal that the exploitation of environmental constraints is a key structural aspect for the robustness and versatility of human grasping behavior. We analyze 3,400 human grasping trials with 17 subjects grasping 25 objects to show that viewing environmental constraint…
We present an integrated approach for perception of unknown articulated objects. To robustly perceive objects and understand interactions, our method tightly integrates pose tracking, shape reconstruction, and the estimation of their kinematic structure. The key insight of our method is that these s
The effectiveness of robot interaction depends on the robot's ability to perform task-relevant actions and on the degree to which it is able to predict the outcomes of these actions. In this paper we argue that the two learning problems - learning actions and learning forward models - must be tightl…
We describe the winning entry to the Amazon Picking Challenge. From the experience of building this system and competing in the Amazon Picking Challenge, we derive several conclusions: 1) We suggest to characterize robotic systems building along four key aspects, each of them spanning a spectrum of…
Soft pneumatic hands offer the advantage of intrinsic mechanical compliance. We argue that to fully leverage the compliance available in soft pneumatic actuators, they should be controlled using air mass rather than position or force, as is customary in most research in soft robotics. We propose an…
We present a method for multi-class segmentation from RGB-D data in a realistic warehouse picking setting. The method computes pixel-wise probabilities and combines them to find a coherent object segmentation. It reliably segments objects in cluttered scenarios, even when objects are translucent, re…
As a first step towards transferring human grasping capabilities to robots, we analyzed the grasping behavior of human subjects. We derived a taxonomy in order to adequately represent the observed strategies. During the analysis of the recorded data, this classification scheme helped us to obtain a…
We present an incremental method for motion generation in environments with unpredictably moving and initially unknown obstacles. The key to the method is its incremental nature: it locally augments and adapts global motion plans in response to changes in the environment, even if they significantly…
There is strong evidence that robustness in human and robotic grasping can be achieved through the deliberate exploitation of contact with the environment. In contrast to this, traditional grasp planners generally disregard the opportunity to interact with the environment during grasping. In this pa…
The ability to selectively stiffen otherwise compliant soft actuators increases their versatility and dexterity. We investigate granular jamming and layer jamming as two possible methods to achieve stiffening with PneuFlex actuators, a type of soft continuum actuator. The paper details five designs…