PARTS: Policy Adaptation with RL on Targeted Subtasks

From Pretraining to ProficiencyReal-World Subtask RL for Long-Horizon Manipulation
with Minimal Human Intervention

Sichang Su1Benjamin Yang2Zhiyun Deng1Boyuan Liang3
Yip Fun Yeung2Zelin Wang2Lingfeng Sun2

1 UT Austin2 Autel US3 UC Berkeley

Corresponding authorsSichang Su (sichang_su@utexas.edu)Lingfeng Sun (lingfengsun1996@gmail.com)
01 OVERVIEW

Targeted practice for long-horizon manipulation

Research overview 1:57Open video

A pretrained robot policy may perform most of a long-horizon task yet fail repeatedly at a few critical subtasks. PARTS (Policy Adaptation with RL on Targeted Subtasks) concentrates real-world practice on these bottlenecks, reusing the behaviors the base policy already performs reliably.

The base policy remains frozen and supplies nominal actions throughout execution. Lightweight residual policies learn targeted adjustments from local subtask rewards, providing useful feedback even when complete-task successes are rare. Online RL alternates with success-reweighted retraining, and improved residuals return to the robot to collect further experience. Humans identify bottlenecks during setup and provide reward checks or physical resets where needed; training rollouts require no human action corrections. Across bimanual YAM and single-arm Franka tasks, these local improvements translate into better performance in complete-task evaluations.

02 METHOD

From bottlenecks to complete tasks

Task identification RL training Full-task evaluation

PARTS architecture (v2): setup and real-world RL loop Target subtask identification and setup Real-world RL on targeted bottlenecks Human: identify bottlenecks from failures ✗ ✗ ✗ Task decomposition Bottleneck selection Subtask 1 Subtask 1 Subtask 2 Bottleneck 1 Subtask 3 Subtask 3 Subtask 4 Bottleneck 2 Human-specified contracts Coding agents Policy selector Success verifier Auto-reset human feedback VLM evidence Offline demos SFT Frozen base policy nominal Residual policy k residual Policy selector Real robot at Reset auto / human Success verifier sparse local reward rk Online TD3+BC actor + twin critics Replay buffer per bottleneck 𝒟k Retrain successes + ρ failures rk update θk redeploy Full-task policy inference during evaluation Base Base + residual … … Base Full-task selector verifier
01

Identify where the base policy fails

Humans review real-robot rollouts to identify the behaviors that prevent complete-task success.

Modules appear in arrow order. Earlier modules stay visible.Open full figure

LEGO and cable rewards are automatic; earbud rewards combine verifier outputs with human checks. The robot resets the scene when feasible, with human assistance otherwise. RL rollouts use no human action corrections.

Figure 2 The original PARTS architecture, revealed module by module along its arrows. View the paper figure

03 THE POLICY IN ACTION

Policy Rollouts

Autonomous execution across three real-world manipulation tasks.

Real-world demonstrations

SUPPLEMENTARY VIEWS

Three-camera earbud rollout

The same earbud rollout from the head and both wrist cameras. On-video labels indicate when the base policy acts alone (VLA) and when a learned residual refines a bottleneck (RL0, RL1, or RL2).

RL0 Case preparation
Open the lid and hold the case as upright as possible, with the earbud slots facing upward.
RL1 First earbud insertion
Align and insert the first earbud into its charging slot.
RL2 Second earbud insertion
Align and insert the second earbud into the remaining charging slot.
Head cameraOpen video
Left wrist cameraOpen video
Right wrist cameraOpen video
04 LEARNING THROUGH PRACTICE

Training

Subtask training from three synchronized cameras: head, left wrist, and right wrist. On-video labels distinguish VLA, RL, and reset phases. VLA and RL execution is autonomous at 1× speed.

Earbud insertion

First earbud insertion · RL1

Human Reset Mode

Repeated insertion practice, with human resets between attempts.

Open video · 3:23

LEGO grasping

Right grasp · RL1   /   Left grasp · RL2

Auto Reset Mode

Right- and left-grasp policies practice repeated pickups, with automatic resets between attempts.

Open video · 2:30
05 FROM SUCCESS TO EFFICIENCY

Reducing Task Completion Time

Full-task performance on the Franka cable unplug-and-plug task, evaluated over 20 trials per method.

MethodSFTDSRLEXPO-FTRLTPARTS Ours
Full-task success rate ↑50%10 / 2065%13 / 2020%4 / 2070%14 / 2095%19 / 20
Mean full-task
completion time ↓		35.8 s21.0 s41.1 s24.0 s20.1 s

Successful rollouts

Select a method to watch one successful episode.

Each video shows an individual successful rollout; its duration may differ from the mean completion time above. Open video

06 REAL-WORLD TRAINING PROTOCOLS

Human Involvement

Who supplies corrections, rewards, and resets during real-world RL?

PARTS at a glance. Expand to compare 10 other real-world RL methods.

MethodHuman Intervention During RL RolloutsReward ModeReset Mode
PARTS OursNo corrections or handoffsSelectors route the base policy and residuals automatically.Automatic*Automatic on LEGO and cable; human checks of verifier labels on earbuds.Automatic*Automatic on LEGO only; human resets for earbuds and cable.

* Task-dependent. Automatic reset is used only for LEGO; reward and reset protocols vary by task as detailed above.

Earbud reward checks

In our experiments, the VLM verifier (Gemini-3.7-flash) made errors distinguishing fully seated from unseated earbuds inside the case. We therefore use human checks of its reward labels for earbuds.

Earbud and cable resets

We use human resets for earbuds and cable insertion because restoring the initial task configuration is difficult for the robot. In particular, extracting earbuds from the case is challenging for our YAM gripper.