Robot Perception

Finding objects in clutter and correcting perception failures during robot operation.

My work on robot perception asks whether a robot can find the right object in a cluttered scene and recover when its first prediction is wrong. I study few-shot object detection, language-guided grounding, and ways for a person to correct a model while the robot is still working. This matters for manipulation: the robot needs a reliable target before it can act.

Progress log

2026 — iTeach: correcting failures in the field

In iTeach, we studied how a person near the robot can correct a segmentation failure as it happens. The person rearranges objects to expose a clear view, then uses gaze and voice to label one frame through a mixed-reality headset. SAM 2 propagates that label through the short RGB-D recording, and the robot fine-tunes its perception model with the new examples. The revised paper reports improved segmentation and downstream pick-and-place results.

2026 — L2G: finding novel object instances

L2G addresses detection when only a few images of the target object are available. Instead of relying on object proposals, it matches local image patches to the templates, filters candidate points, and uses them to prompt a segmentation model for a complete object mask. Our RSS 2026 paper evaluates the method in cluttered open-world scenes.

March 2026 — RPX benchmark development

I began organizing Robot Perception X (RPX) to test perception methods across scene conditions and deployment constraints. The evaluation covers visual grounding, object tracking, depth estimation, relative camera pose, and novel-view synthesis. Camera-pose ground truth was still being refined in March, and the benchmark remains under development.

2025 — Multimodal reference visual grounding

In MRVG, we studied how language and reference images can identify a target among similar objects. The work introduced a dataset and a method combining few-shot detection with language-based matching. The paper describes both.

Examples from this direction

L2G: local matches to object masks

In a crowded scene, only a small part of the target may be visible. L2G matches that part to template images, then uses the matches to prompt a complete instance mask.

L2G comparison of proposal-based detection and local-to-global instance detection

L2G compares object proposals with local matching followed by mask reconstruction. Select the image for the full-size figure.

iTeach: learning from a failure

When the robot misses an object, a person can expose it and label one frame. iTeach uses that brief correction to update the robot’s perception model while it is still in the scene.

iTeach workflow from a robot perception failure to human annotation and model fine-tuning

A perception failure becomes a short human teaching interaction and new training data.

These projects extend my interest in perception to longer activities, where small errors can affect what happens next. In dexterous manipulation, the same issue arises when a robot must identify an object before handling it.