102 accepted papers
Deep stereo disparity estimation has long been dominated by a \textbf{matching-centric paradigm}, built on constructing cost volumes and iteratively refining local correspondences. Despite its success, this paradigm exhibits an intrinsic vulnerability: visual ambiguities from occlusion or non-Lamber…
Vision-Language Model (VLM) is now an important component to enable robust robot manipulation. Yet, using it to translate human instructions into an action-resolvable intermediate representation often needs a tradeoff between VLM-comprehensibility and generalizability. Inspired by context-free gramm
With the current surge in spatial reasoning, researchers have made significant progress in understanding indoor scenes, but still struggle with more diverse applications. This paper aims to advance all-scale spatial reasoning by tackling two key challenges: 1) the heavy reliance on indoor 3D scans a…
Shape control of deformable objects under both rotational and translational deformations is important for versatile robotic applications. However, deformation control with full 6-degree-of-freedom (DoF) manipulation is an open problem, since modeling and describing rotational deformations lead to si
For objects with complex topological and geometrical features, stochastic topological grasping can be executed without the necessity for feedback or precise planning. However, this grasping method has two significant limitations. First, the technique's effectiveness is reduced when interacting with
Monocular and stereo depth estimation offer complementary strengths: monocular methods capture rich contextual priors but lack geometric precision, while stereo approaches leverage epipolar geometry yet struggle with ambiguities such as reflective or textureless surfaces. Despite post-hoc synergies,…
Flexible continuum robots operating in constrained and dynamic environments face significant challenges, especially when interacting with uncertain and potentially unknown conditions. Traditional model-based methods face significant difficulties due to the inherent nonlinearities and uncertainties i
Observing that the key for robotic action planning is to understand the target-object motion when its associated part is manipulated by the end effector, we propose to generate the 3D object-part scene flow and extract its transformations to solve the action trajectories for diverse embodiments. The
In this paper, we introduce a novel image-goal navigation approach, named RFSG. Our focus lies in leveraging the fine-grained connections between goals, observations, and the environment within limited image data, all the while keeping the navigation architecture simple and lightweight. To this end,
Accurate and robust simultaneous localization and mapping (SLAM) is crucial for autonomous mobile systems, typically achieved by leveraging the geometric features of the environment. Incorporating semantics provides a richer scene representation that not only enhances localization accuracy in SLAM b
Multi-view stereo (MVS) implicitly encodes photometric and geometric cues into the cost volume for multi-view correspondence matching, transferring insufficient geometric cues essential to depth estimation and reconstruction. This paper proposes GE-MVS, a novel multi-view stereo network with geometr
Model Predictive Control (MPC) enables agile and robust locomotion in quadruped robots but is sensitive to model uncertainties and environmental variations. This paper presents a Tube-Based Robust MPC (TR-MPC) framework for quadruped locomotion under uncertainties, modeled as parameter mismatches an
In the era of Large Language Models (LLMs), embodied artificial intelligence presents transformative opportunities for robotic manipulation tasks. Ultrasound imaging, a widely used and cost-effective medical diagnostic procedure, faces. challenges due to the global shortage of professional sonograph
Exploring the limits of quadruped robot agility, particularly in the context of rapid and real-time planning and execution of omnidirectional jump trajectories, presents significant challenges due to the complex dynamics involved, especially when considering significant impulse contacts. This paper…
Soft tissue tracking is crucial for computer-assisted interventions. Existing approaches mainly rely on extracting discriminative features from the template and videos to recover corresponding matches. However, it is difficult to adopt these techniques in surgical scenes, where tissues are changing…
Model Predictive Control (MPC) relies heavily on the robot model for its control law. However, a gap always exists between the reduced-order control model with uncertainties and the real robot, which degrades its performance. To address this issue, we propose the controller of integrating a data-dri…
Efficiency and reliability are critical in robotic bin-picking as they directly impact the productivity of automated industrial processes. However, traditional approaches, demanding static objects and fixed collisions, lead to deployment limitations, operational inefficiencies, and process unreliabi
The development of robotic systems for palletization in logistics scenarios is of paramount importance, addressing critical efficiency and precision demands in supply chain management. This paper investigates the application of Reinforcement Learning (RL) in enhancing task planning for such robotic
Given a set of outlier-contaminated 2D–3D line correspondences between the scene and a captured image, we aim to recover the absolute camera pose. This is a fundamental problem in computer vision and robotics, for which many methods have been developed and shown impressive performance, but they fail
Automated laparoscope field of view (FoV) control in minimal invasive surgery (MIS) poses challenges, as existing solutions failed to address dynamic surgical FoV requirements across different phases and they neglected the misorientation effect or potential obstacles during the control process which
Stereo matching is a core task for many computer vision and robotics applications. Despite their dominance in traditional stereo methods the hand-crafted Markov Random Field (MRF) models lack sufficient modeling accuracy compared to end-to-end deep models. While deep learning representations have gr…
Robotic bin packing is a challenging task, requiring compactly packing objects in a container and also efficiently performing the computation, such that the robot arm need not wait too long before taking action. In this work, we introduce PPN-Pack, a novel learning-based approach to improve the effi
Photometric bundle adjustment (PBA) is widely used in estimating the camera pose and 3D geometry by assuming a Lambertian world. However, the assumption of photometric consistency is often violated since the non-diffuse reflection is common in real-world environments. The photometric inconsistency s…
Given an input set of 3D point pairs the goal of outlier-robust 3D registration is to compute some rotation and translation that align as many point pairs as possible. This is an important problem in computer vision for which many highly accurate approaches have been recently proposed. Despite their…
Recently, fiber optic sensors such as fiber Bragg gratings (FBGs) have been widely investigated for shape reconstruction and force estimation of flexible surgical robots. However, most existing approaches need precise model parameters of FBGs inside the fiber and their alignments with the flexible r…
In this work, we present a multi-stage pipeline that aims to accurately predict suction grasps for objects with varying properties in cluttered and complex scenes. Existing methods face difficulties in generalizing to unseen objects and effectively handling noisy depth/point cloud data, which often
Object pose estimation plays a crucial role in robotic manipulation, however, its practical applicability still suffers from limited generalizability. This paper addresses the challenge of generalizable object pose estimation, particularly focusing on category-level object pose estimation for unseen…
Recent advancements toward perception and decision-making of flexible endoscopes have shown great potential in computer-aided surgical interventions. However, owing to modeling uncertainty and inter-patient anatomical variation in flexible endoscopy, the challenge remains for efficient and safe navi…
Scene reconstructions are often incomplete due to occlusions and limited viewpoints. There have been efforts to use semantic information for scene completion. However, the completed shapes may be rough and imprecise since respective methods rely on 3D convolution and/or lack effective shape constrai…
Task automation of surgical robot has the potentials to improve surgical efficiency. Recent reinforcement learning (RL) based approaches provide scalable solutions to surgical automation, but typically require extensive data collection to solve a task if no prior knowledge is given. This issue is kn…
Localization in a pre-built map is a basic technique for robot autonomous navigation. Existing mapping and localization methods commonly work well in small-scale environments. As a map grows larger, however, more memory is required and localization becomes inefficient. To solve these problems, map s…
Offline evolutionary-based methodologies have supplied a successful motion planning framework for the quadrupedal jump. However, the time-consuming computation caused by massive population evolution in offline evolutionary-based jumping framework significantly limits the popularity in the quadrupeda…
Shape from Polarization (SfP) aims to recover surface normal using the polarization cues of light. The accuracy of existing SfP methods is affected by two main problems. First, the ambiguity of polarization cues partially results in false normal estimation. Second, the widely-used assumption about o…
Estimating depth from four large field of view (FoV) cameras has been a difficult and understudied problem. In this paper, we proposed a novel and simple system that can convert this difficult problem into easier binocular depth estimation. We name this system OmniVidar, as its results are similar t…
Instance segmentation is a long-standing task for supporting robotic bin picking. However, objects of diverse classes can be closely packed with occlusions in cluttered and chaotic scenes, hence, even recent methods could have difficulty in locating clear and precise boundaries to distinguish nearby…
Few-Shot 3D Point Cloud Object Detection (FS3D) is a challenging task, aiming to detect 3D objects of novel classes using only limited annotated samples for training. Considering that the detection performance highly relies on the quality of the latent features, we design a VAE-based prototype learn…
Robotic bin packing is very challenging, especially when considering practical needs such as object variety and packing compactness. This paper presents SDF-Pack, a new approach based on signed distance field (SDF) to model the geometric condition of objects in a container and compute the object pla…
Most existing methods for category-level pose estimation rely on object point clouds. However, when considering transparent objects, depth cameras are usually not able to capture high-quality data, resulting in point clouds with severe artifacts. Without a complete point cloud, existing methods are…
Automatic laparoscope motion control is fundamentally important for surgeons to efficiently perform operations. However, its traditional control methods based on tool tracking without considering information hidden in surgical scenes are not intelligent enough, while the latest supervised imitation…
We present a generic and robust sim-to-real deep-learning-based framework, namely S2R-Pick, for fast and accurate object recognition and localization in industrial robotic bin picking. Unlike existing works designed for general everyday environments, objects for industrial bin picking are often text
"This paper studies category-level object pose estimation based on a single monocular image. Recent advances in pose-aware generative models have paved the way for addressing this challenging task using analysis-by-synthesis. The idea is to sequentially update a set of latent variables,e.g., pose, s…
This paper presents an optimal motion planning framework to generate versatile energy-optimal quadrupedal jumping motions automatically (e.g., flips, spin). The jumping motions via the centroidal dynamics are formulated as a 12-dimensional black-box optimization problem subject to the robot kino-dyn…
Given a set of putative 3D-3D point correspondences, we aim to remove outliers and estimate rigid transformation with 6 degrees of freedom (DOF). Simultaneously estimating these 6 DOF is time-consuming due to high-dimensional parameter space. To solve this problem, it is common to decompose 6 DOF, i…
Estimating precise metric depth and scene reconstruction from monocular endoscopy is a fundamental task for surgical navigation in robotic surgery. However, traditional stereo matching adopts binocular images to perceive the depth information, which is difficult to transfer to the soft robotics-base…
Robots have been used extensively in the battle against the COVID-19 pandemic since its outbreak. One prominent direction is the use of robots for swab sampling, which not only solves the shortage of medical staffs, but also prevents them from being infected during face-to-face sampling. However, a
Smart laparoscope motion control for adjusting surgical field-of-view is an increasingly hot topic in robot-assisted surgery. Previous off-the-shelf methods have been conducted in reactive ways which heavily rely on human input signals, e.g., gaze or voice, thus cannot avoid cognitive burdens to sur
Manipulation of deformable objects is an essential task in surgery, the textile industry, and household tasks, such as washing, hanging, and folding clothes. However, current studies on fabric manipulation, have rarely considered the scenes in which rigid objects have to be wrapped by fabrics. This
In this work, we study the problem of how to leverage instructional videos to facilitate the understanding of human decision-making processes, focusing on training a model with the ability to plan a goal-directed procedure from real-world videos. Learning structured and plannable state and action sp
Instance segmentation is an important task for supporting robotic grasping in auto-store scenarios. Accurate segmentation usually relies on the quantity and quality of available annotated training data. However, it requires tremendous cost to obtain these labels. In this work, without requiring any…
"6D object pose estimation is important for robotic bin-picking, and serves as a prerequisite for many downstream industrial applications. However, it is burdensome to annotate a customized dataset associated with each specific bin-picking scenario for training pose estimation models. In this paper,…
Industrial bin picking is a challenging task that requires accurate and robust segmentation of individual object instances. Particularly, industrial objects can have irregular shapes, that is, thin and concave, whereas in bin-picking scenarios, objects are often closely packed with strong occlusion.…
Laparoscopic Field of View (FOV) control is one of the most fundamental and important components in Minimally Invasive Surgery (MIS), nevertheless the traditional manual holding paradigm may easily bring fatigue to surgical assistants, and misunderstanding between surgeons also hinders assistants to…
Embryos/oocytes vitrification is an essential cryopreservation technique in IVF (in vitro fertilization) clinics. The reliable and effective transferring of embryos/oocytes is crucial to the subsequent steps in the whole procedure of vitrification. After each transferring, the straw needs to be repl…
This paper presents a comprehensive inertial aided 3D LiDAR SLAM system with hybrid geometric primitives in large-scale environments, including a tightly-coupled LiDAR-Inertial-Odometry (LIO), a global mapping module supported by learning-based loop closure detection and a sub-maps matching algorith…
Existing vanishing point (VP) estimation methods rely on pre-extracted image lines and/or prior knowledge of the number of VPs. However, in practice, this information may be insufficient or unavailable. To solve this problem, we propose a network that treats a perspective image as input and predicts…
Given a set of putative 2D-2D line correspondences, we aim to identify correct matches. Existing methods exploit the geometric constraints. They are only applicable to structured scenes with orthogonality, parallelism and coplanarity. In contrast, we propose the first approach suitable for both stru…
Surgical instrument segmentation in robot-assisted surgery (RAS) - especially that using learning-based models - relies on the assumption that training and testing videos are sampled from the same domain. However, it is impractical and expensive to collect and annotate sufficient data from every new…
This letter presents a practical method for parameter estimation of a full-size industrial car-like tractor. With only on-vehicle sensors including three encoders, an IMU and a 3D LiDAR, we estimate the effective radius of rear wheels, ratio between the steering wheel angle and road wheel angle, the
Automatic surgical gesture recognition is fundamentally important to enable intelligent cognitive assistance in robotic surgery. With recent advancement in robot-assisted minimally invasive surgery, rich information including surgical videos and robotic kinematics can be recorded, which provide comp…
Autonomous surgical execution relieves tedious routines and surgeon’s fatigue. Recent learning-based methods, especially reinforcement learning (RL) based methods, achieve promising performance for dexterous manipulation, which usually requires the simulation to collect data efficiently and reduce t…
Soft robots have been applied widely to various constrained scenarios due to the advantages over traditional rigid manipulators such as softness, deformability and adaptability to constrained surroundings. To make full use of this merit, this paper proposes a method that integrates collision detecti…
Surgical knot tying is one of the most fundamental and important procedures in surgery, and a high-quality knot can significantly benefit the postoperative recovery of the patient. However, a longtime operation may easily cause fatigue to surgeons, especially during the tedious wound closure task. I…
Three-dimensional reconstruction of dynamic objects is important for robotic applications, for example, the robotic recognition and manipulation. In this paper, we present a novel 3D surface reconstruction method for moving objects. The proposed method combines the spatial-multiplexing and time-mult
Cooperative adaptive cruise control (CACC) of intelligent vehicles contributes to improving cruise control performance, reducing traffic congestion, saving energy and increasing traffic flow capacity. In this paper, we resolve the CACC problem from the viewpoint of synchronization control, our main…
Robot-assisted vitreoretinal surgery can filter surgeons' hand tremors and provide safe, accurate tool manipulation. In this paper, we report the design, optimization, and evaluation of a novel tilt mechanism for a new Steady-Hand Eye Robot (SHER). The new tilt mechanism features a four-bar linkage…
This paper presents a novel dataset targeting three types of challenging environments for autonomous driving, i.e., the industrial logistics environment, the undulating hill environment and the mixed complex urban environment. To the best of the author’s knowledge, similar dataset has not been publi…
Atlanta world holds for the scenes composed of a vertical dominant direction and several horizontal dominant directions. Vanishing point (VP) is the intersection of the image lines projected from parallel 3D lines. In Atlanta world, given a set of image lines, we aim to cluster them by the unknown-b…
Conventional robot hand-eye calibration methods are impractical for localizing robotic instruments in minimally-invasive surgeries under intra-corporeal workspace after preoperative set-up. In this letter, we present a new approach to autonomously calibrate a robotic instrument relative to a monocul
This paper presents a novel solution for online trajectory planning of a full-size tractor-trailers vehicle composed of a car-like tractor and arbitrary number of passive full trailers. The motion planning problem for such systems was rarely addressed due to the complex nonlinear dynamics. A simulat…
In this paper, we propose a dynamic state estimation framework for lateral control of a heavy tractor-trailers system using only mass-produced low-cost sensors. This issue is challenging since the lateral velocity of the lead tractor is difficult to measure directly. The performance of existing dyna…
This letter reports a general path following framework for the tractor-trailers system in Global Positioning System (GPS)-denied environments. Compared to existing methods, this approach prioritizes a robust, cost-optimized, and easy-to-implement solution. First, to achieve accurate path following,
Given a set of 3D-to-2D point correspondences corrupted by outliers, we aim to robustly estimate the absolute camera pose. Existing methods robust to outliers either fail to guarantee high robustness and efficiency simultaneously, or require an appropriate initial pose and thus lack generality. In c…
This paper addresses the problem of self-supervised video representation learning from a new perspective -- by video pace prediction. It stems from the observation that human visual system is sensitive to video pace, g, slow motion, a widely used technique in film making. Specifically, given a video…
Three-dimensional vision plays an important role in robotics. In this paper, we present a 3D surface reconstruction scheme based on combination of stereo matching and pattern projection. A two-step matching scheme is proposed to establish reliable correspondence between stereo images with high compu
In this paper, we study the cooperative path planning and motion coordination problems of the multi-robot system with large number of robots, aiming for practical applications in robotic warehouses and automated transportation systems. Particularly, we solve the life-long planning problem and guaran…
Inherent compliance plays an enabling role in soft robots, which rely on it to mechanically conform to the environment. However, it also limits the payload of the robots. Various variable stiffness approaches have been adopted to limit compliance and provide structural stability, but most of them ca…
While the mechanism of Rope-Climbing provides much flexibility, it opens up challenges to the development of the controller for Robotic Manipulator installed on Rope-Climbing robot(RCR), which is called Rope-Climbing Robot Manipulator(RCRM) here. In particular, the deformable nature of the rope resu…
In robotic-assisted laparoscopic surgery, the first assistant (FA) stands at the bedside assisting the intervention, while the surgeon sits at the console teleoperating the robot. Tasks for the FA include navigating new instruments into the surgeon's field-of-view and passing in or retracting materi…
We present an autonomous manipulation approach for tissues with anisotropic deformation behavior using a continuum manipulator. The key feature of our vision-based study is an online learning and estimation method, which makes its implementation independent of any prior knowledge about the deformati
Large demand for robotics and automation has been reflected in the sanding works, as current manual operations are labor-intensive, without consistent quality, and also subject to safety and health issues. While several machines have been developed to automate one or two steps in the sanding works,…
A major issue for needle insertion into soft tissue during suturing is the induced tissue deformation that hinders the minimization of tip-target positioning error. In this letter, we present a new robot control framework to solve target deviation by integrating active deformation control. We charac
Wall polishing is a typical and essential procedure in the interior renovation. However, such works are mainly carried out by humans, which have the disadvantages of low efficiency, inconsistent quality, and issues of safety and health. A new vision-based impedance controller is proposed for polishi…
Point cloud based place recognition is still an open issue due to the difficulty in extracting local features from the raw 3D point cloud and generating the global descriptor, and it's even harder in the large-scale dynamic environments. In this paper, we develop a novel deep neural network, named L…
A wide range of man-made environments can be abstracted as the Atlanta world. It consists of a set of Atlanta frames with a common vertical (gravitational) axis and multiple horizontal axes orthogonal to this vertical axis. This paper focuses on leveraging the regularity of Atlanta world for monocul…
3D lines in structured environments encode particular regularity like parallelism and orthogonality. We leverage this structural regularity to estimate the absolute and relative camera poses. We decouple the rotation and translation, and propose a novel rotation estimation method. We decompose the a…
Existing works on control of tractor-trailers systems only consider the kinematics model without taking dynamics into account. Also, most of them treat the issue as a pure control theory problem whose solutions are difficult to implement. This paper presents a trajectory tracking control approach fo…
The image lines projected from parallel 3D lines intersect at a common point called the vanishing point (VP). Manhattan world holds for the scenes with three orthogonal VPs. In Manhattan world, given several lines in a calibrated image, we aim at clustering them by three unknown-but-sought VPs. The…
Place recognition and loop-closure detection are main challenges in the localization, mapping and navigation tasks of self-driving vehicles. In this paper, we solve the loop-closure detection problem by incorporating the deep-learning based point cloud description method and the coarse-to-fine seque…
Most existing controllers for Car-Like Mobile Robots (CLMR) are designed to handle dynamic effects by decoupling speed and steering controls, also assume that full states are accessible, which are unrealistic for real-world applications. This paper presents a combined speed and steering control syst…
This paper focuses on the localization and mapping problem on ground vehicles using odometric and monocular visual sensors. To improve the accuracy of vision based estimation on ground vehicles, researchers have exploited the constraint of approximately planar motion, and usually implemented it as a…
In this paper, we propose a novel 3D laparoscopic imaging system based on stereo-photogrammetry which is assisted by projecting patterns on the tissue surface. The proposed laparoscopic imaging system has three optic channels, two of which are responsible for stereo vision feedback and the other one…
Robot malfunction is inevitable in practical applications of the robot formation control due to uncontrolled crashing, system malfunction or communication loss. In this paper, we study the synchronous formation control problem in the presence of robot malfunctions. Our main idea is to improve the ne…
We present a generic data-driven method to address the problem of manipulating a three-dimensional (3-D) compliant object (CO) with heterogeneous physical properties in the presence of unknown disturbances. In this study, we do not assume a prior knowledge about the deformation behavior of the CO an
The ability of rope-climbing robots in aloft operation is limited by its self-supporting and locomotion ability. In many applications, a given task is also too complex to be achieved by a single rope-climbing robot acting alone. The solution of multiple rope-climbing robots can overcome the limitati…
Emerging applications of robotic manipulation of deformable objects have opened up new challenges in robot control. While several control techniques have been developed to manipulate deformable objects, the performance of existing methods is commonly limited by two issues: 1) implicit assumption tha…
The fast expanding 3C (Computer, Communication, and Consumer electronics) manufacturing leads to a high demand on the fabrication of USB cables. While several commercial machines have been developed to automate the process of stripping and soldering of USB cables, the operation of manipulating USB w…
Most existing trajectory tracking controllers are based on non-skidding and non-slipping assumptions, also assume that full states are accessible, which is unrealistic for real-world applications due to tire-road interaction. This paper presents a novel vision-based approach to achieve high performa…
In this paper, we address the problem of the image-based 3D pose computation of a semi-circle suturing needle using monocular image feedback for laparoscopy. We propose a constrained two-degree-of-freedom (2-DOF) geometry-based modelling method to parametrise the needle's 6-DOF pose, including depth…
In this paper, we present a new robotic uterine positioner for total laparoscopic hysterectomy. The robot is designed to actively position the patient's uterus during surgery, a lengthy and tedious task that is traditionally performed by a human assistant. Safety is simply the most important concern…
In this paper, we address the positioning problem of remote centre of motion (RCM) mechanisms with uncalibrated image feedback from a monocular camera. Nowadays, RCM mechanisms are widely used in minimally invasive robotic surgery due to their ability to distally rotate a tool around a fixed entry p…
In this paper, we propose a novel design of compliant safe joint, which has flexibility when the work load exceeds a predefined threshold. The compliance is generated by a spring. We design a special transmission mechanism to convert axial motion into circumferential motion such that the linear comp…
This paper presents the development of an endoscope manipulator with passive and active structures for functional endoscopic sinus surgery (FESS). The 5-DoF passive structure has three translations and two rotations (T3R2) that allows the surgeon to manually place the endoscope near to the entry poi…