arXiv
Abstract
Multi-agent manipulation naturally produces multiple task-driven viewpoints: every arm carries a wrist camera and moves through the scene while acting. Yet these observations are typically underutilized, and active perception in manipulation is still often treated as requiring a dedicated sensing agent. We introduce MAAP (Multi-Agent Active Perception), in which every arm is dual-purpose: it executes manipulation actions and, through the wrist camera it carries, simultaneously serves as a moving viewpoint for the team. We pair this with RAIL (Role-Aware Imitation Learning), a controller that predicts each arm’s current role alongside its action chunk and conditions action generation on it, representing role-dependent actions within one network. Across four simulated tasks, widening the perception regime lifts average success from 56.5% with a fixed camera to 62.5% with one active wrist view and 70.0% with all of them, while MAAP+RAIL reaches 79.2%. RAIL’s additional gain is concentrated on the three-arm Microwave task, where success rises from 47% to 82% on identical multi-wrist inputs. On a dual-arm platform, MAAP+RAIL succeeds in 14 of 20 placement trials compared with 0 of 20 for fixed-view ACT. Collaborative manipulation can thus serve as an active perception mechanism in its own right.
MAAP: Multi-Agent Active Perception
Use the collaborating arms to both act and see.
A fixed camera can lose sight of an object behind a robot arm, inside a container, or beneath a shelf. In MAAP, each arm carries a wrist camera, and the policy receives all of the team’s wrist views. As the arms move, their cameras provide complementary observations: a target hidden from one view may become visible from another.
Each arm can manipulate an object or move to obtain a useful view for a teammate. These responsibilities can change during an episode. For example, opening a microwave makes its interior visible, shifting the useful viewpoint from the door handle to the object inside. Every wrist camera continues to contribute observations, regardless of the arm’s current role.
Simulation Demonstrations
The policy uses all wrist-camera viewpoints. Arms that are not currently manipulating an object actively move to observe the scene and provide useful views for the team.
Stack Cube
Pick from Pot
Pick from Microwave
Place on Cart
RAIL: Role-Aware Imitation Learning
Learn when each arm should manipulate and when it should seek a view.
RAIL implements this coordination within an imitation-learning policy. From the shared observations, it predicts each arm’s manipulation or perception role at every step of an action chunk, then uses those predictions to condition the actions. The role assignments are updated when the policy replans from new observations.
Share observations
Combine the team’s wrist views while keeping every arm available to act.
Infer each arm’s role
Predict manipulation and perception roles for every step of the action chunk.
Condition the action
Use the predicted roles to modulate features and mix role-specific actions.
RAIL Real-World Rollout
The role labels shown during inference are predicted by the model and change as the task progresses.
Experimental Results
Success rates (%) in simulation and on a dual-arm robot.
| Task | Fixed view GV | Single wrist active perception SAV | All wrists MAAP | All wrists + roles MAAP+RAIL |
|---|---|---|---|---|
| Real Robot | ||||
| Place Blocked Cube | 0 | — | — | 70 |
| Simulation | ||||
| Stack Cube | 8 | 37 | 34 | 35 |
| Pick from Pot | 86 | 99 | 99 | 100 |
| Pick from Microwave | 32 | 14 | 47 | 82 |
| Place on Cart | 100 | 100 | 100 | 100 |
| Simulation Average | 56.5 | 62.5 | 70.0 | 79.2 |
Simulation: 100 episodes per configuration, one training run; MAAP and MAAP+RAIL use the same wrist views. Real robot: 20 trials per condition; fixed-view ACT and MAAP+RAIL differ in both visual input and controller. — denotes not evaluated. Simulation Average is the unweighted mean over the four simulation tasks. Bold indicates the highest value in each row.