LENS Helps Robots Ignore the Clutter They Do Not Need

A robot asked to move one object across a crowded table does not necessarily need a perfect physical model of every bowl, block, and distant obstacle. Yet many planning and control systems inherit the whole scene anyway. Each extra object can add collision checks, contact combinations, or visual distractions, even when it has nothing to do with the goal.

Researchers at the University of Pennsylvania built LENS to reduce that burden before a robot acts. The system uses a vision-language model to decide which objects can be pruned from a task representation and which connected objects can be merged into one functional group. It then sends the simplified scene to an existing planner, controller, or vision-language-action model.

The idea sounds like deleting information, because it is. The important part is that LENS can revise its choices after a timeout or execution error instead of assuming its first abstraction was correct.

Prune what is irrelevant, merge what moves together

LENS receives a task description, the objects in the scene, and information about the downstream robot system. Its vision-language model makes two kinds of reductions.

Pruning marks objects the robot does not need to manipulate. A planning system can leave those objects fixed for collision checking, while a learned visual policy can have them removed from its image through segmentation and inpainting. Merging treats several objects as a single unit when their internal arrangement is not important to the task, such as a tray and the objects stacked on it.

Those decisions can depend on meaning as well as distance. A nearby object may be irrelevant, while a more distant target or obstacle still matters. That is where the language model is intended to improve on a fixed geometric rule.

It can also be wrong. If the reduced scene produces an infeasible plan, exceeds a time limit, or triggers a controller error, LENS adds that failure to its prompt and asks for a revised abstraction. The hardware tests allowed up to three of these feedback loops.

One front end, three kinds of robot system

The researchers connected LENS to task-and-motion planning, contact-implicit model predictive control, and a vision-language-action policy. These systems normally represent scenes in different ways, so the integration changes by stack while preserving the same basic reduction loop.

For task-and-motion planning, pruned objects remain collision geometry but cannot become actions, and stable stacks can become one object. In heavy clutter, this reduces the branching search that otherwise makes a plan difficult to find.

For model-based pushing control, merging and pruning reduce the number of bodies, possible contact pairs, and optimization variables. The simulation comparison is the clearest scaling result. LENS-C3+ succeeded in 39 of 45 trials, while the full-scene baseline succeeded in 17 of 30. At six objects, the baseline took about 1,000 seconds; at seven, it exceeded 4,000 seconds. LENS stayed between roughly 40 and 135 seconds across the tested clutter levels.

On a real robot performing planar pushing, both the pruning-only and pruning-plus-merging versions reached an 80 percent cumulative success rate. Successes occurred across all three attempts, showing that feedback sometimes recovered after an aggressive first reduction omitted a relevant contact.

Removing visual distractions helps learned policies too

LENS also filtered images before passing them to a vision-language-action model. In Spatial LIBERO simulation tests, adding clutter reduced average task success from 0.85 to 0.50 across 100 episodes and ten tasks. Filtering the clutter with LENS raised it to 0.69.

The hardware test used open-vocabulary detection, segmentation, and inpainting rather than privileged simulator labels. A robot had to select one of six fruit plush toys and put it in a bowl. Across four reported fruit tasks with ten trials each, the cluttered baseline succeeded only twice and failed every banana, pear, and apple trial. LENS succeeded 18 times, including seven of ten apple trials.

That does not recover the uncluttered performance, and the paper is candid about the remaining gap. The abstraction can still be imperfect, the underlying policy still faces distribution shift, and the system does not learn from mistakes across separate runs.

A smaller world model can be more useful

LENS is a research preprint, not evidence that a general household robot can safely tidy an arbitrary room. Its experiments are tabletop tasks, and its feedback uses high-level failure signals rather than a precise diagnosis of what went wrong.

The broader result is nevertheless useful: adding more observations is not always the route to better robot behavior. A task-conditioned world model can be smaller than the visible world and still retain what the robot needs. LENS shows that the same principle can improve symbolic search, physics-based control, and learned action policies without replacing any of them.