Markus Vincze

MIND - Multi-Feature Implicit Neural Descriptors for Robotic Surface Processing of 3D Objects with Variations in Geometry

The recent shift from mass production to mass personalization leads to a production environment in which workpieces have a high degree of geometric variations. The robotic process automation in such high-mix low-volume environments poses significant challenges since predetermined robot programs are …

SGFAM: Semantic and Geometric Features Aggregation for Dense Shape Matching in Generalizable Robotic Manipulation

To automate processes like polishing or cleaning at scale, robots must be able to adapt learned skills to new object instances without manual reprogramming. Applications requiring tool-surface interactions face a significant challenge in transferring manipulation strategies to novel objects due to s

SilRef: Joint Visual Silhouette and Tactile Pose Optimization for Transparent Object Manipulation

Transparent objects are ubiquitous in laboratory automation settings, as liquids need to be visually controlled regularly. Automating laboratory processes would make the creation of small-batch medication feasible, thus making more personalized and better-targeted treatments more accessible. However

SilRef: Joint Visual Silhouette and Tactile Pose Optimization for Transparent Object Manipulation

MIND - Multi-Feature Implicit Neural Descriptors for Robotic Surface Processing of 3D Objects With Variations in Geometry

The recent shift from mass production to mass personalization leads to a production environment in which workpieces have a high degree of geometric variations. The robotic process automation in such high-mix low-volume environments poses significant challenges since predetermined robot programs are

ODYSSEE: Oyster Detection Yielded by Sensor Systems on Edge Electronics

Oysters are a vital keystone species in coastal ecosystems, providing significant economic, environmental, and cultural benefits. As the importance of oysters grows, so does the relevance of autonomous systems for their detection and monitoring. However, current monitoring strategies often rely on d

EdgeSoil 2.0 – Soil Analyzer Using Convolutional Neural Network and Camera Imaging for Agricultural Robotics

Soil is the most important building element of agriculture and its analysis is crucial for healthy plants and a high crop yield. But apart from its importance, soil analysis is a tedious and time-consuming task. This paper presents EdgeSoil 2.0, a non-invasive, accurate, and real-time robotic system…

NRDF - Neural Region Descriptor Fields as Implicit ROI Representation for Robotic 3D Surface Processing

To automate 3D surface processing across diverse category-level objects it is imperative to represent process-related region of interest (P-ROI), which is not obtained with conventional keypoint or semantic part correspondences. To resolve this issue, we propose Neural Region Descriptor Fields (NRDF

ReFlow6D: Refraction-Guided Transparent Object 6D Pose Estimation via Intermediate Representation Learning

Transparent objects are ubiquitous in daily life, making their perception and robotics manipulation important. However, they present a major challenge due to their distinct refractive and reflective properties when it comes to accurately estimating the 6D pose. To solve this, we present <italic xmln

ZS6D: Zero-shot 6D Object Pose Estimation using Vision Transformers

As robotic systems increasingly encounter complex and unconstrained real-world scenarios, there is a demand to recognize diverse objects. The state-of-the-art 6D object pose estimation methods rely on object-specific training and therefore do not generalize to unseen objects. Recent novel object pos…

3D-DAT: 3D-Dataset Annotation Toolkit for Robotic Vision

Robots operating in the real world are expected to detect, classify, segment, and estimate the pose of objects to accomplish their task. Modern approaches using deep learning not only require large volumes of data but also pixel-accurate annotations in order to evaluate the performance and therefore…

GigaDepth: Learning Depth from Structured Light with Branching Neural Networks

"Structured light-based depth sensors provide accurate depth information independently of the scene appearance by extracting pattern positions from the captured pixel intensities. Spatial neighborhood encoding, in particular, is a popular structured light approach for off-the-shelf hardware. However…

Robust Sim2Real 3D Object Classification Using Graph Representations and a Deep Center Voting Scheme

While object semantic understanding is essential for service robotic tasks, 3D object classification is still an open problem. Learning from artificial 3D models alleviates the cost of the annotation necessary to approach this problem, but today’s methods still struggle with the differences between

Object Learning for 6D Pose Estimation and Grasping from RGB-D Videos of In-hand Manipulation

Object models are highly useful for robots as they enable tasks such as detection, pose estimation and manipulation. However, models are not always easily available, especially in real-world domains of operation such as peoples’ homes. This work presents a pipeline to generate high-quality object re…

PyraPose: Feature Pyramids for Fast and Accurate Object Pose Estimation under Domain Shift

Object pose estimation enables robots to understand and interact with their environments. Training with synthetic data is necessary in order to adapt to novel situations. Unfortunately, pose estimation under domain shift, i.e., training on synthetic data and testing in the real world, is challenging…

ReAgent: Point Cloud Registration Using Imitation and Reinforcement Learning

Point cloud registration is a common step in many 3D computer vision tasks such as object pose estimation, where a 3D model is aligned to an observation. Classical registration methods generalize well to novel domains but fail when given a noisy observation or a bad initialization. Learning-based me…

Addressing the Sim2Real Gap in Robotic 3-D Object Classification

Object classification with 3D data is an essential component of any scene understanding method. It has gained significant interest in a variety of communities, most notably in robotics and computer graphics. While the advent of deep learning has progressed the field of 3D object classification, most

Neural Object Learning for 6D Pose Estimation Using a Few Cluttered Images

Recent methods for 6D pose estimation of objects assume either textured 3D models or real images that cover the entire range of target poses. However, it is difficult to obtain textured 3D models and annotate the poses of objects in real scenarios. This paper proposes a method, Neural Object Learnin…

Robust and Efficient Object Change Detection by Combining Global Semantic Information and Local Geometric Verification

Identifying new, moved or missing objects is an important capability for robot tasks such as surveillance or maintaining order in homes, offices and industrial settings. However, current approaches do not distinguish between novel objects or simple scene readjustments nor do they sufficiently deal w…

Unsupervised Domain Adaptation Through Inter-Modal Rotation for RGB-D Object Recognition

Unsupervised Domain Adaptation (DA) exploits the supervision of a label-rich source dataset to make predictions on an unlabeled target dataset by aligning the two data distributions. In robotics, DA is used to take advantage of automatically generated synthetic data, that come with “free” annotation

VeREFINE: Integrating Object Pose Verification With Physics-Guided Iterative Refinement

Accurate and robust object pose estimation for robotics applications requires verification and refinement steps. In this work, we propose to integrate hypotheses verification with object pose refinement guided by physics simulation. This allows the physical plausibility of individual object pose est

EasyLabel: A Semi-Automatic Pixel-wise Object Annotation Tool for Creating Robotic RGB-D Datasets

Developing robot perception systems for recognizing objects in the real world requires computer vision algorithms to be carefully scrutinized with respect to the expected operating domain. This demands large quantities of ground truth data to rigorously evaluate the performance of algorithms. This p…

Multi-Task Template Matching for Object Detection, Segmentation and Pose Estimation Using Depth Images

Template matching has been shown to accurately estimate the pose of a new object given a limited number of samples. However, pose estimation of occluded objects is still challenging. Furthermore, many robot application domains encounter texture-less objects for which depth images are more suitable t…

Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation

Estimating the 6D pose of objects using only RGB images remains challenging because of problems such as occlusion and symmetries. It is also difficult to construct 3D models with precise texture without expert knowledge or specialized scanning devices. To address these problems, we propose a novel p…

Recurrent Convolutional Fusion for RGB-D Object Recognition

Providing robots with the ability to recognize objects like humans has always been one of the primary goals of robot vision. The introduction of RGB-D cameras has paved the way for a significant leap forward in this direction thanks to the rich information provided by these sensors. However, the rob

Robust 3D Object Classification by Combining Point Pair Features and Graph Convolution

Object classification is an important capability for robots as it provides vital semantic information that underpin most practical high-level tasks. Classic handcrafted features, such as point pair features, have demonstrated their robustness for this task. Combining these features with modern deep…

ScalableFusion: High-resolution Mesh-based Real-time 3D Reconstruction

Dense 3D reconstructions generate globally consistent data of the environment suitable for many robot applications. Current RGB-D based reconstructions, however, only maintain the color resolution equal to the depth resolution of the used sensor. This firmly limits the precision and realism of the g…

Action Selection for Interactive Object Segmentation in Clutter

Robots operating in human environments are often required to recognise, grasp and manipulate objects. Identifying the locations of objects amongst their complex surroundings is therefore an important capability. However, when environments are unstructured and cluttered, as is typical for indoor huma…

Multi-View 3D Entangled Forest for Semantic Segmentation and Mapping

Applications that provide location related services need to understand the environment in which humans live such that verbal references and human interaction are possible. We formulate this semantic labelling task as the problem of learning the semantic labels from the perceived 3D structure. In thi…

Recognizing Objects in-the-Wild: Where do we Stand?

The ability to recognize objects is an essential skill for a robotic system acting in human-populated environments. Despite decades of effort from the robotic and vision research communities, robots are still missing good visual perceptual systems, preventing the use of autonomous agents for realwor…

Towards Autonomous Auto Calibration of Unregistered RGB-D Setups: The Benefit of Plane Priors

In the last few years novel color and depth (RGB-D) sensors have greatly pushed robot perception. To enable a precise pixel-wise fusion of color and depth information good calibration is needed. The calibration determines the intrinsic parameters, the extrinsic parameters, and corrects for depth err…

Autonomous Learning of Object Models on a Mobile Robot

In this article, we present and evaluate a system, which allows a mobile robot to autonomously detect, model, and re-recognize objects in everyday environments. While other systems have demonstrated one of these elements, to our knowledge, we present the first system, which is capable of doing all o

RGB-D fusion enhancement by mode filter for surfel cloud segmentation

This paper presents an algorithm for surfel color and position enhancement from RGB-D data acquired across multiple image frames. Surfel-based reconstruction algorithms associate each RGB-D frame pixel to a surfel in the model. As the reconstruction progresses, surfel color and position are the aver…

Calibration and correction of vignetting effects with an application to 3D mapping

Cheap RGB-D sensors are ubiquitous in robotics. They typically contain a consumer-grade color camera that suffers from significant optical nonlinearities, often referred to as vignetting effects. For example, in Asus Xtion Live Pro cameras the pixels in the corners are two times darker than those in…

Enhancing Semantic Segmentation for Robotics: The Power of 3-D Entangled Forests

We present a novel, fast, and compact method to improve semantic segmentation of three-dimensional (3-D) point clouds, which is able to learn and exploit common contextual relations between observed structures and objects. Introducing 3-D Entangled Forests (3-DEF), we extend the concept of entangled

Viewpoint Evaluation for Online 3-D Active Object Classification

We present an end-to-end method for active object classification in cluttered scenes from RGB-D data. Our algorithms predict the quality of future viewpoints in the form of entropy using both class and pose. Occlusions are explicitly modeled in predicting the visible regions of objects, which modula

Fast and accurate normal estimation by efficient 3d edge detection

Accurate surface normal computation is one of the most basic and important tasks for 3d perception. While much progress has been made in speeding up normal estimation algorithms and improving their accuracy, a significant inaccuracy still remains even with modern implementations, which is the correc…

Fast semantic segmentation of 3D point clouds using a dense CRF with learned parameters

In this paper, we present an efficient semantic segmentation framework for indoor scenes operating on 3D point clouds. We use the results of a Random Forest Classifier to initialize the unary potentials of a densely interconnected Conditional Random Field, for which we learn the parameters for the p…

RGB-D object modelling for object recognition and tracking

This work presents a flexible system to reconstruct 3D models of objects captured with an RGB-D sensor. A major advantage of the method is that unlike other modelling tools, our reconstruction pipeline allows the user to acquire a full 3D model of the object. This is achieved by acquiring several pa…

Saliency-based object discovery on RGB-D data with a late-fusion approach

We present a novel method based on saliency and segmentation to generate generic object candidates from RGB-D data. Our method uses saliency as a cue to roughly estimate the location and extent of the objects present in the scene. Salient regions are used to glue together the segments obtained from…

Temporal integration of feature correspondences for enhanced recognition in cluttered and dynamic environments

We propose a method for recognizing rigid object instances in RGB-D point clouds by accumulating low-level information from keypoint correspondences over multiple observations. Compared to existing multi-view approaches, we make fewer assumptions on the recognition problem, dealing with cluttered an…