Share from 0:00
0:00
/

How to Make Better Robot Policies with Dense Subtask Labels?

we are interviewing Jade Choghari from Scale AI about how they labelled thousands of hours of data to make more robust robot policies!

most robot datasets give the robot one instruction for a whole episode, like “clean the table”, and then a few minutes of motion.

the model has to figure out by itself where one step ends and the next begins which is hella hard.

I had the pleasure of chatting with jade choghari from scale AI, who walks us through FineART, a 1,700 hour two-arm robot dataset where every single step is labeled in plain words by a human (534k labels, no VLM in the loop!)

these subtask labels do a lot of heavy lifting to shape a proper policy.

without them the model can’t even reliably tell the left bin from the right one, even with a big vision-language backbone behind it!

with them, the steps it learned in words carry over when transfered to a completely different robot, to the point where it pulls off a long breakfast task it never practiced there and even reuses an “adjust” move it only ever saw on the previous robot!

very fun chat, we also get into how scale ai actually collects this stuff behind the scenes which is as impressive as the result themselves!

👉 Jade Website: https://jadechoghari.github.io/
👉 Jade GitHub: https://github.com/jadechoghari
👉 FineART (paper): https://arxiv.org/abs/2609.36416
👉 FineART-VLA (model): https://huggingface.co/lerobot/fineart_vla_base

Check out Arcee (American open research lab and sponsor of this interview):
♥️ https://www.arcee.ai/
Also trinity large thinking is available on open router:
♥️ https://openrouter.ai/arcee-ai/trinity-large-thinking#providers
♥️ and check out the NAC harness https://www.arcee.ai/blog/nac

Table of Content:
- 00:00:00 - what is FineART?
- 00:03:55 - my man jade choghari at ScaleAI SF HQ
- 00:06:34 - highschool days to machine learning research engineering
- 00:10:10 - the waterloo magic touch 🪄
- 00:10:50 - how did you get into lerobot?
- 00:12:46 - what drove jade into ai research?
- 00:15:01 - FineART: Data + VLA walkthrough
- 00:16:08 - “Divide each difficulty into as many parts as is feasible and necessary to resolve it” Rene Descartes
- 00:17:30 - how do you decide what a subtask should be?
- 00:21:56 - why full 100% human organic??
- 00:24:35 - dataset distribution
- 00:26:28 - how in the world did you pull off 100% human annotation in there man
- 00:31:05 - FineART-VLA policy + loss function
- 00:36:18: FineART-VLA interesting emerging behavior
- 00:42:00 - why does knowledge insulation protect the VLM?
- 00:45:15 - you think we could relax knowledge insulation once the expert is up to speed?
- 00:47:10 - what type of steps does the model get wrong?
- 00:51:31 - what context does the model get for each task?
- 00:55:20 - how are you handling error in the teleoperation setup?
- 01:01:00 - how many review passes does a label go through before it’s considered good?
- 01:04:00 - the surprising result in the breakfast task
- 01:07:40 - what makes skills transfer across robots just work?
- 01:09:50 - where did the model failed?
- 01:11:25 - does the model get lost on long tasks?
- 01:12:35 - why did ABC collapse while FineART kept climbing the benchmark?
- 01:14:17 - what’s next for FineART?
- 01:19:08 - how should you mix sim + teleop + human video data?
- 01:22:28 - is the VLM backbone here to stay?
- 1:26:25 - what is jade excited about these days?

Discussion about this video

User's avatar

Ready for more?