LeRobot: Open-Source End-to-End Robot Learning with Steven & Pepijn from Hugging Face

there is this kind of renaissance in robotics / physical ai at the moment and it's very stimulating.

things finally just work ™️

I had the pleasure of chatting with two experts from HuggingFace immersed in this environment steven palma and pepijn kooijmans from the LeRobot team about this renaissance.

the gist of the mission of LeRobot is that robotics is very different from software/LLM in the sense that if you wanna start tinkering with robot you need to shell out quite a lot of money to get something working and handle a maddening stack of complexity.

so the team at LeRobot is trying to level up the playing field by basically doing what the Transformers library did for NLP but for robotics.

it's one library that takes you from plugging in a cheap arm like the SO-101/microduck/etc. to recording data, to training a policy (ACT, SmolVLA, all the way up to ports of the π series), to running it straight on the real robot.

there are 90,000+ LeRobot datasets on the hub with more than 200 terabytes of robot data (and it's growing)!

in this interview we get into:
- how to get started in robotics
- why real world is much better than sim
- what porting a VLA like π0.5 actually looks like
- why almost no data is bad data anymore as long as it's labeled right
- Steven's essay on why open robotics makes us safer
- how frontier models like Astra drive robots by writing classical inverse kinematics code
- whether VLAs are dead now that world action models are here
- and why robotics is just much harder than language!

amazing session btw did my best to but a whole bunch of reference straight into the video hope you enjoy!

useful links

👉 Steven Hugging Face: https://huggingface.co/imstevenpmwork
👉 Pepijn Hugging Face: https://huggingface.co/pepijn223
👉 LeRobot (GitHub): https://github.com/huggingface/lerobot
👉 LeRobot: An Open-Source Library for End-to-End Robot Learning (main paper): https://arxiv.org/abs/2602.22818
👉 Why I Believe Open Robotics Makes Us Safer, Not Less Safe (Steven's essay): https://imstevenpmwork-open-robotics-blogpost.hf.space/
👉 FineART (paper): https://arxiv.org/abs/2609.36416

Check out Arcee (American open research lab and sponsor of this interview):
♥️ https://www.arcee.ai/
Also check out trinity large thinking:
♥️ https://www.arcee.ai/blog/trinity-large-thinking
♥️ and check out the NAC harness https://www.arcee.ai/blog/nac

here is the full table of content btw:

- 00:00:00 - to play with AI for robots you need a robot
- 00:01:41 - sponsor: Arcee and the NAC harness
- 00:03:43 - meet Steven and Pepijn from LeRobot
- 00:04:02 - what did you do before getting into robotics?
- 00:05:59 - how would you advise someone to get into robotics?
- 00:09:07 - how do I get my 7 year old into robotics?
- 00:11:55 - what is LeRobot? the end-to-end robot learning pipeline
- 00:15:32 - is LeRobot the Transformers library for robotics?
- 00:17:38 - open source hardware and making robots cheap
- 00:21:22 - why real world first instead of simulation first?
- 00:24:32 - the LeRobot community: hobbyists, startups, labs + the global hackathon
- 00:27:35 - you don't need a frontier budget to do robotics research
- 00:29:41 - how do research labs and industry connect?
- 00:32:35 - no single thing will solve robotics: data, model size, latency, factory wifi
- 00:37:08 - is Astra doing GPT-3 level zero-shot generalization now?
- 00:39:59 - frontier models going back to classical robotics with inverse kinematics scripts
- 00:42:53 - RT-1 tokenized actions and System 1 / System 2 setups
- 00:45:53 - any plan to make robotics harnessing easier?
- 00:49:14 - is LeRobot a platform or just a very good library?
- 00:50:40 - the 3 principles of LeRobot
- 00:53:08 - what does porting a VLA like π0.5 look like?
- 00:55:40 - community optimizations vs the paper + you can't vibe test a robot
- 00:58:31 - any bias toward some policies over others?
- 00:59:57 - survival of the fittest for policies
- 01:03:42 - could the 88k LeRobot datasets enable broad generalization?
- 01:05:52 - a little book in the big library that everyone can read
- 01:08:01 - what makes good demonstration data vs bad data?
- 01:09:21 - almost no data is bad data if it's labeled right (feat. FineART)
- 01:13:01 - record as many sensor modalities as you can
- 01:16:46 - core argument of Steven's essay on open robotics and safety
- 01:23:54 - put the bottleneck on hardware, not on research
- 01:26:51 - the wifi analogy
- 01:28:17 - are we close to finding the scaling law for robotics?
- 01:31:25 - could there be multiple scaling laws for different components?
- 01:33:53 - how many scenarios do you need to test to call a model general?
- 01:38:08 - is the VLM backbone a crutch or a crucial component?
- 01:42:08 - are VLAs dead now that world action models are here?
- 01:44:01 - dead for who? use whatever works (XGBoost is still king sometimes)
- 01:45:54 - is data the answer or are we still waiting for our GPT moment?
- 01:47:06 - the GPT moment was distribution + instructability
- 01:51:16 - how different is RL post-training in robotics vs language models?
- 01:55:00 - what's next for you and for LeRobot?

enjoy!!

Full Transcript

with full references!!!

00:00:00 – Robotics for the Price of a Laptop

Yacine Mahdid

If you want to play with AI for text today, all you need is a laptop. If you want to play with AI for robots, you need a robot. And for a long time, that meant something costing around $100,000.

So today I’m with Pepijn and Steven from LeRobot at Hugging Face, which is an open-source community effort to make robotics much more accessible. LeRobot is basically what Hugging Face’s Transformers library was for NLP, but for robotics. It’s one library that takes you all the way from plugging in a cheap arm, to recording some data, to training a policy, and then literally running inference straight on the real robot.

The LeRobot paper, describing one open-source library for end-to-end robot learning, from hardware to deployment. (source)

It ships everything from small models like ACT and SmolVLA all the way up to frontier models like the π series.

ACT (Action Chunking with Transformers), one of the small policies LeRobot ships. (source)

SmolVLA, Hugging Face’s small vision-language-action model for affordable robotics. (source)

π0, Physical Intelligence’s vision-language-action flow model, the frontier end of what LeRobot supports. (source)

The main vision behind this library is that the real magic in robotics is in the real world, and the only way to get a lot of real-world data is to make robotics cheap enough that the whole community can start to collect it. And it’s actually working. There are now around 88,000 LeRobot datasets on the Hub (I think it’s even higher than that now), and it’s growing. That’s more than 200 terabytes of robotics data.

The number of LeRobot datasets on the Hugging Face Hub over time, climbing steeply. (source)

So in this interview, we’ll cover LeRobot, of course, but I’ve also asked them for a few of their opinions on robotics in general. Stuff like why planning might already be mostly solved and where the real bottleneck in robotics might be. Why almost no data is bad data anymore, as long as it’s labeled. How frontier models are actually controlling these robots without being trained on them. Whether VLAs are dead now that world models are something that actually works. And why robotics is just harder than language.

I’ll link the LeRobot repo and their ICLR paper down below. It’s a great companion to this conversation. Enjoy.

00:03:43 – Robotics Is a Galaxy of Planets

Steven Palma

Hello everyone, my name is Steven. I’m a robotics engineer here at Hugging Face. I’ve been working on LeRobot almost from its very beginning, or well, around two years already. I’ve always been in the robotics space, so I’m very excited to be here. And thanks, Yacine, for the invitation.

Yacine Mahdid

Cool. I just want to understand a bit more: what did you do before getting into robotics?

Steven Palma

I like to see robotics as this galaxy, right, which is composed of different planets. Even before being here at Hugging Face, I was going around different planets every time, because robotics is a composition of systems. That’s what I’m trying to say.

Steven’s picture of robotics as a galaxy made of many different planets.

For some time I was doing mostly automation or electromechanics. For some other time I was doing networking. Some other time I was doing mostly perception. Now we’re doing physical AI, which is essentially machine learning, right? But I’ve had the time to spend a couple of years on each planet to explore the field in general.

And before robotics, I think I’ve always done it, even when I was studying. It was just called differently. Sometimes it’s called mechanics, sometimes it’s called electronics, sometimes you just call it computer science. But ultimately it’s all part of this galaxy I was talking about.

Yacine Mahdid

True. And even when you look at a VLA, it’s a composition of a bunch of stuff, right? You have a VLM over there, you have an expert for the action, and you have all of these little tricks that you learn from different subsets of machine learning that get composed together. On top of that, you also have the physical constraint of latency, so the inference needs to happen at a time that actually makes sense. Today it really looks like a composition. Cool, glad to have you here. Pepijn, you can go ahead.

Pepijn Kooijmans

Yeah, so I’m Pepijn. Thanks for having me. I’ve also been working on LeRobot for almost two years now. We’re doing a bunch of different things, all the way from hardware to software and ML. It’s been amazing until now. I think it’s what Steven says: it’s so amazing to work on so many different things.

Yacine Mahdid

How would you advise somebody who’s interested in robotics? I’ve seen this problem with pure machine learning too, where people say they want to get into deep learning, but they have no clue where to start, because machine learning is also a composition of many things. And with machine learning it’s a bit easier, because you don’t necessarily have hardware to handle. So what would you tell somebody who’s interested in robotics to get started?

Pepijn Kooijmans

The most important thing is to just start, right? But that’s true with everything, I think. You have so many different components in robotics. So I think the first thing in any technology, especially nowadays with AI, is mapping out the space. Which components are there? What are the edges of the components? Which things overlap? And trying to create this mental map of where everything should be.

And especially now with AI, you have the opportunity to go into the details and the depth of every little part, and to actually test whether your landscape makes sense and whether things actually work together the way you thought.

Yacine Mahdid

Yeah, true. I think the mapping is a bit difficult for some, but you could just use AI and get a pretty good map.

Steven Palma

Yacine, if I might interrupt, just to add a little bit to what Pepijn said, because I think it’s very important. Going back to the example I gave at the beginning about robotics being this galaxy: you eventually have to pick your planet, right? You have to pick your flavor, or where to start.

The same galaxy of robotics, now with the advice to pick one planet to start from.

Do you want to explore hardware first? Maybe you want to explore electronics first. Or do you want to explore the software stack, which can be decomposed into different things? It can be computer vision, it can be perception, it can even be machine learning today. So it’s this umbrella that covers a lot of topics, and it’s hard to find one single thing that will help you explore all of them at the same time.

I think it’s important, as Pepijn said, to just start out with something, knowing that it’s going to take some time to explore these different things. It’s important to pick one flavor, the one you’re most interested in, and take it from there. You’re eventually going to move to another planet and explore new things within this robotics umbrella or galaxy.

00:08:28 – Optimizing for the Wow Moment

Yacine Mahdid

Funny enough, I have a 7-year-old, and I can actually show you guys. It’s this dinky robot that we built together. It’s all busted now; it’s literally holding on by the tread. It’s messed up. But they love that stuff so much. We did that over the summer, and right now he’s asking me, “Okay, can we do more robots?”

He didn’t know we were at the point where there are humanoid robots. For him there were just drones and little mobile tank-type robots. So for somebody who doesn’t have the ability to create a map and all that, how would you suggest I get this kid into robotics? He’s been asking me consistently for two weeks now.

Steven Palma

My take on that, especially when it comes to kids and teenagers, is that you want to avoid overloading them with a bunch of information and all the challenges and difficulty of the field. You want to optimize for the wow moment. You want to optimize for those little wows they can have, so that it triggers their curiosity to continue their own journey and their own path for learning.

So for kids and teenagers, I think it’s very important to have these little robots or prototypes like the one you’ve shown. But we also have things like Reachy Mini, Microduck and the SO-100s, which are very accessible because they optimize for the wow moment. You’ve never seen it, you get super curious, you get more interested in it, and that’s how your journey starts.

Yacine Mahdid

Yeah, it’s a good point. This one was a pain, because we had to assemble everything and he was right next to me while I was screwing stuff in wrong. But they still liked the fact that they could control it. I explained the code to them and said, “I just need to get this into the firmware and then it can run.” And when it started running, they were like, “Okay, yeah, this is cool.”

I actually tried to buy a Microduck, but I think the shipping time is 4 to 6 months. So I got it ordered, and I just need to find something else in the meantime to get him going before he loses interest.

Pollen Robotics’ Microduck, the small walking robot Yacine ordered for his son. (source)

Steven Palma

I think the original objective for Microduck was to ship it before Christmas, and there are some units that are going to go out before Christmas. But there were so many orders and so much demand that we were like, okay, we can’t keep promising that, because it’s not going to be possible.

The Microduck pre-order page on the Pollen Robotics store. (source)

But Microduck is just one example of how hungry people are in general to get to know these systems and build some context and intuition about how they work, in a very accessible manner. It doesn’t need to be super intimidating, it doesn’t need to be super hard, it doesn’t even need to be capable. Microduck only walks around, and maybe it can pick something up. It’s just a matter of putting a device out there for people to play with.

Yacine Mahdid

Yeah, true. I think I’m going to get either the Reachy Mini or, I think it’s called LeKiwi, the one that moves around. He doesn’t even know the Microduck exists, so I’m going to wait for that a bit, give him something else in the meantime, and we’re going to build it out.

Reachy Mini, Hugging Face and Pollen Robotics’ small open-source desktop robot. (source)

LeKiwi, a low-cost mobile robot with an arm on a wheeled base. (source)

Cool, perfect. So I think it’s a good time to explain to folks what LeRobot is. I think we can use the French pronunciation here.

00:12:23 – LeRobot, From Motors to Deployment

Steven Palma

So LeRobot is essentially the robotics initiative we have here at Hugging Face, where we asked how we could bring all the innovations happening in AI to the physical world. That means essentially everything happening in physical AI, but in a more accessible manner. This is how LeRobot was born. LeRobot is a fully open-source project that makes this end-to-end learning pipeline accessible to everyone in the community.

By pipeline I mean the technologies we see today. All the crazy things we see on social media, a robot doing a backflip or cooking a meal or doing tasks that were traditionally very, very challenging, more or less follow the same path. First you need hardware, and you need to control that hardware, which is the robot. Then you go and collect some datasets, which are the ingredients you’re going to use to cook your model. The model is the intelligence, the part where we apply all the innovations coming from the LLM space, but at the robot level. Once you have your model, you run it on your robot, and you also evaluate it in simulation on some of the benchmarks.

So in a nutshell, LeRobot is this software library, this framework, that walks you through all these different steps. We give you a very accessible robot to start with, because you’re going to need something that moves; you can’t get into robotics without any hardware. Then we take care of how you teleoperate it and how you move it. From the moment you connect it to the moment the robot is actually moving takes maybe 10 minutes, and you don’t need four PhDs and a full month to understand how to move the robot.

After that, you start collecting datasets. Sorry if I’m using a lot of examples, but I like to think of it as a restaurant. The ingredients used to prepare your meal are the datasets. The recipe is the model itself. And how you deploy and evaluate is the restaurant experience.

So in LeRobot, you collect your datasets, and we give you all the tooling for that. Then you pick the policy, the model you want to train, which might be a VLA, a world model, or who knows what comes next. Even reward models, or reinforcement learning as well. Then you deploy it on your robot to see it actually accomplish the task you wanted to automate. So again, LeRobot is this library that walks you through all these steps so that you can build some context about how these technologies work. The idea is that tomorrow, when you see the robot doing the backflip again, you already have some context about how that was done. And if you want to give it a try, you can just go ahead.

00:15:30 – The Transformers for Robotics

Yacine Mahdid

It reminded me a lot of when Hugging Face released the Transformers library. I see exactly the same similarity. Before, everything was fragmented and everybody was doing whatever. But then the field coalesced around it, and the dataset format followed afterward. I feel like this is the same thing, but condensed, and you guys are doing all of it at the same time. Was this the main idea behind LeRobot?

Pepijn Kooijmans

Yeah, that’s a similar analogy. We try to be the Transformers for robotics, but in general, just create all the tools people might need to use this end-to-end learning for robotics, and anything that might fall under it. I really want to build useful tools for everyone, basically, and unify the community.

The LeRobot paper, which presents the library as one open-source stack for end-to-end robot learning. (source)

Steven Palma

And going back to your point about datasets and the field being fragmented, that’s something we also experienced. If a research lab came out with some exciting research, it was very hard for someone else to replicate it. That challenge came from different angles: maybe the hardware, maybe the dataset format, maybe the policy semantics or the model semantics they were using.

So with LeRobot, we try to unify that so the breakthroughs are a bit more streamlined. If someone does very cool research, maybe they’re very good at collecting data or at creating a new academic benchmark, then by having it on common ground, which is now LeRobot, it’s way easier for us to level up the field overnight. If Pepijn makes a breakthrough at the policy level tomorrow, which is actually not that far from being true, then someone working on the benchmark can immediately leverage that discovery. It’s like we’re all speaking the same language, which is of course very beneficial for the field in general.

Yacine Mahdid

I think this is what really attracted me to this project. You have the data component, the software, the middleware component, but then you have the literal hardware. You guys are actually building the hardware, putting it out there and making sure it’s as cheap as possible so that as many people from the open-source community as possible can start tinkering with it. And you can see it: I was reading the paper, and you could see it in the number of datasets that were uploaded.

From the LeRobot paper: downloads, datasets and episodes on the Hub over time, broken down by robot type; look at how steeply the community uploads climb. (source)

I think it’s a pretty cool way to get this engine started, standardizing and centralizing a bit more the work everybody is doing, and having this open-source situation in the robotics, mechanical engineering and electrical engineering type of field. Generally speaking, and I work with a lot of these guys, it’s not really like that. There’s not that much open sharing. The tools in manufacturing are usually closed source, people keep their things to themselves, and I can see them all reinventing a bunch of the same stuff. So I think it’s pretty cool; you’re starting to get this momentum here.

Pepijn Kooijmans

Historically, hardware has been less open source in general, and especially in robotics. There used to be a lot of individual robotics builders, hobbyists building their own robot. But what the SO-101 and the SO-100 did well, I think, is put a bit of a bigger organization behind it, so these robots could be manufactured at scale very cheaply and have some continuity. So with hardware, it’s very nice that we could fill some gaps there and also be really open.

The open SO-100 and SO-101 arms from TheRobotStudio, whose designs are published openly so they can be built cheaply at scale. (source)

Steven Palma

And to complement that take: if you want to experiment today with LLMs, with the typical AI we know, the only thing you need is a computer, a keyboard and a screen, and you’re ready to go. But if you want to experiment in physical AI, you need more than just a computer. You need hardware, which was my previous point. And if that hardware costs $100,000, then no one is going to experiment with it. So our take was: let’s make the robot very, very basic and very, very accessible, and it will serve as a vehicle, a landing point for people to experiment with all the rest of the features.

From the LeRobot paper: low-cost open robots set against industrial arms, and how cheap hardware lets data collection spread across the community. (source)

Of course, eventually you’ll realize the SO-100 is lacking here or there for your task, but you’ve already had a taste of all the steps I was talking about at the beginning. It’s like, okay, I already know how the dataset works, I already know how the policy works. You saw the robot moving, and you start imagining: what if I get better hardware? What if I get a better dataset? What if I get better compute? So the importance of accessible hardware and of being open source is that it gives people a channel they can tap into to experiment with the rest of the physical AI innovations.

00:21:19 – Why the Real World Comes First

Yacine Mahdid

I think it’s pretty cool. I’ve seen somewhere that a key point you emphasize in LeRobot is that it’s real world first, instead of simulation first. I think I know why, but can you explain the rationale behind it?

Pepijn Kooijmans

I think the magic is in the real world. We love to see real things moving, and especially seeing the robots move for the first time is very magical, and figuring out that it learned this just from images. Aside from that, the real world is where we eventually want to apply this technology. If you do robotics, it needs to be applied in the real world. So if you really want to make an impact, you also want to encourage the community to record real-world data, to give a bit back to the community. And the only way to do that is to make robots cheap, so we can get a lot of people in the community, so we can get a lot of data, and so they also do it in the real world.

Steven Palma

Absolutely. And to echo what Pepijn said, there are different things at play. Pepijn mentioned the wow moment, which is of course very exciting, not only when you first see the robot moving but all the following times too, when you actually see it accomplishing the little task you trained it for. That’s a wow moment full of dopamine; it’s just very magical to see.

The second thing is the importance of having such accessible hardware. It’s so accessible that it’s even easier to try things out with the physical robot than to think about something more complex. If the hardware costs $200, you just plug it into your computer and that’s it. You don’t need to think about anything else. You already have the hardware, you already have the task you want to automate right there, and you’re ready to go.

And the third thing is that being physical first, as Pepijn was saying, makes sense because that’s where the robots will eventually live, so you need to experiment with the robots there. But it’s also because simulation has traditionally had its own challenges, and solving those challenges in simulation doesn’t mean you’ve solved them in the physical world. Maybe it’s not better, but it’s a sensible approach to start right away from the physical side and take it from there.

Yacine Mahdid

That makes a whole bunch of sense. Tackling the hard part head-on is worth it here, because that’s where it’s going to be used at the end of the day. The feedback loop will be there, and people who actually want to do real stuff will gravitate toward that type of ecosystem.

00:24:23 – The Community Behind LeRobot

Steven Palma

Maybe Pepijn also, I think you had more interesting things in your presentation. Maybe you can share yours too.

Pepijn Kooijmans

I think the most important part of LeRobot, which we try to reiterate, is the community, right? The people using it. We’re building this for quite a variety of people. Originally, as I said, you have a lot of hobbyists in robotics, individuals just experimenting with motors, cameras, etc. So that’s a big chunk.

But what you now also see, of course, is that more and more companies are joining and exploring this embodied AI wave. That’s aside from all the researchers we have, all the universities, etc., contributing to the field. We try to really serve all those people. We try to build all the useful tools for them.

And what we hopefully get back is their use and all their feedback. There are a bunch of projects that the community has done. They can hook up their own robots, they can hook up their own models, and they really build on top of what we’ve already done, which is awesome to see.

Steven Palma

And maybe to echo Pepijn’s words even more, we have also realized that LeRobot is not only there for research labs, as I was mentioning at the beginning, to collaborate with each other and so on. It’s also for startups, big enterprises, or even hobbyists, because it serves as this one-stop shop for robot learning. People can come and pick the part of the project they’re most interested in. Someone who is working on data curation might come in from the dataset side. Someone working on the model and intelligence parts may go and collaborate on the model side. And so on.

But the point here is that it is not only the project but the ecosystem, the community, as Pepijn was saying, that is actually fostering the innovation along all these different axes of the project. And it’s not only us from the LeRobot team or Hugging Face. It’s also other enterprises, startups, the community, the hobbyists, all around the world.

Last year we had this global hackathon, in which we were, I don’t have it off the top of my head, but I think more than 50 countries or something like that. In the same weekend, just full of robot people sharing their applications, sharing datasets, collaborating with each other. That’s what makes this project so vibrant: the community is so active, so engaged.

Even yesterday we had the first LeRobot Live, which is our initiative to connect even more closely with the community, where the users of the project can ask us questions and get to know the team. It’s to close the gap between what is being done in the project and what the field and the ecosystem are expecting.

00:27:35 – Why Robot Learning Is Still Open to Everyone

Yacine Mahdid

Yeah. That’s maybe one of the reasons why we’re seeing so much momentum in the robotics space, a kind of renaissance. If you want to do frontier LLM research, good luck, right? Because we have the scaling laws. As soon as the model gets big enough, it’s another beast. It’s not the same as when it’s small. You can still do very interesting research with small models, but at some point, to know more, you just have to spend $500,000 to make a run and check. And you don’t have that budget.

But I feel like with robotics, there’s something about the fact that the data is not internet scale, so you can’t just go YOLO and have a gigantic pre-trained model from the get-go. The models are also not that big. I was looking at the scale of some of them. I think the biggest one was from Google in 2023, like 50B or something like that. But most of them are in the 2 to 9B range. And depending on the task, you might just go with ACT, which is much smaller, and it could just blitz through your thing with its latency.

So it seems like there’s a mix: the data is super hard to get, and you have to get it from the real world. If you just do sim, it’s not the same thing. And the models are not that gigantic. I think it’s Generalist, or, I don’t remember the company, that started to see some scaling laws happening, but it’s still in the earliest stage of that scaling.

Generalist’s GEN-0 blog post, which reports scaling laws for embodied foundation models trained on physical interaction data. (source)

So I feel like a lot of people can participate and make cool stuff without necessarily having a high-end robotic arm that costs a million dollars. You can still do the same thing if you have some of the hardware and enough compute to run it. Do you think that’s a fair assessment? And what have you seen in terms of the different players doing this research: the research labs, the people from industry who literally have manufacturing lines? Do you see some connection there?

Pepijn Kooijmans

The link between industry and research is very important. We also try to build that bridge as much as possible, inviting people from industry over and also looking into as much of the new research being done as we can.

Specifically because everyone uses a computer, right? So it’s quite easy to think about all the use cases for LLMs, especially once they’re there. But for robotics, aside from all the things you have to do in your own home, I think it’s very hard to come up with all the pick-and-place or assembly tasks that people are doing in factories. I’ve visited a fair bit of factories. But at the same time, all the different products in your house were assembled by someone, somewhere.

So if you think about all the potential use cases for robotics, they’re so wide that I think no single researcher could come up with them, create a benchmark for them and try to solve them. Bringing those two together, to actually know what we need to solve, is very important for robotics. The application layer is so wide that it’s very hard to build something that solves all of it if you don’t know what everything is.

Yacine Mahdid

Yeah. And the data is different depending on how you capture it. And you also don’t necessarily need generalist models. For example, in manufacturing, they can do whatever they want in the little cells they have. They can remove all the distractions and just have something very specific. Whereas if it’s in the real world, in your kitchen, with a humanoid robot, those are two different things.

They connect in some sense: you have the latency, there’s some model that will work best, you still have to collect the data and go through this flow. But I feel like there’s less need for a very generalist model that just works everywhere.

Actually, I saw this thing this morning, I think from the meetup in Quebec. They did the first scaling law. But if you look at that scaling law, it goes from 20 million to 8B. It’s not gigantic. It’s still fairly much in the world of, yeah, you could run that. You don’t need some crazy compute somewhere.

00:32:35 – No Single Fix for Robotics

Steven Palma

Yeah, maybe to tie the loop with your original question, Yacine, where we discussed the availability of data in the wild, the size of the model, the capabilities of the hardware, and then the application layer. I think we’re realizing, and this is echoing within the field, that there’s no single thing that, if we solve it, robotics will be solved. There are still multiple challenges, even in the pipeline we were explaining at the beginning.

A lot of people today are focusing on data because it’s one of the most obvious ones. For LLMs we had, and have, as you know, the internet. Everything was digitized. But that’s not the case for physical AI. That’s why there’s so much interest in the data format, the scale, the raw amount of data out there.

But then we also have things like the model. If we come up with an enormous model, you won’t be able to run it locally on your robot. And do you actually want your robot to rely on a Wi-Fi connection to an API endpoint that might be somewhere else?

Yacine Mahdid

Yeah, this is super important. I was in a meeting with a manufacturer, and they were thinking about using a bigger model just to test. The guy was like, “I have a GPU, I can just test it out. I’m gonna go downstairs and test it.” These are boxes that they sell, with a robotic arm in them, a camera and so on, and it’s doing welding and things like that. And the CTO was like, “Why do you want to add a GPU? If you add a GPU there, it’s just gonna go straight into the cost. Can’t you use something smaller?”

So you have that element. And you can’t just assume you’re going to have an internet connection, like you said. Some of these facilities are almost like a Faraday cage. It’s a bunker. It’s very hard just to get a normal signal. So expecting the thing to go out and come back with low enough latency is almost unrealistic.

Steven Palma

Yeah. I think this is rooted in the fact that we are applying the LLM innovations to the physical AI space. And we’re assuming all these different things: that we’ll have access to compute, that we already have the data, and so on. All these assumptions get challenged when you actually try to deploy something in the wild. Oh, you need to collect the data. You need more compute. Oh, but my real-time requirements are different from the LLM space.

If your question about which restaurants you can eat at in Paris takes 1 minute to come back at you, you don’t mind that much. But you cannot afford that type of latency in your robotics application, especially in production. In my personal experience, warehouses and production lines tend to be very, very strict about uptime and efficiency, because that’s their whole business. You want to make sure your cookies keep going out the door on a frequent basis. You can’t afford something that’s not up to standard, and also slower.

Also, from my experience, as you said, as soon as you enter the factory floor and say, “Hey, my little device, which is totally external to your infrastructure, needs Wi-Fi access, either to your infrastructure or out of it,” the guy will tell you, “No, no, no, wait. I’ll have to go and pull in all the security people to make sure your device is validated.” So although we might have a lot of breakthroughs in the robotics space nowadays, there are still a lot of challenging things we need to figure out.

The way I like to see it, AI in general has given us robots that, when you deploy them in a certain application, make that application more flexible. It doesn’t need to be as deterministic and as rigid as it was before, because the robot is more general. But that was only one part of the robot, the part that actually automates the task. You still have all the other sides of the robotics space that you need to take care of.

00:37:06 – Astra and the Ice Cube Test

Yacine Mahdid

Yeah. There’s a community question from Martin Kemka: “Remember a few weeks ago when lots of posts spoke about the GPT-3 moment with zero-shot local action models from a few private startups? Is Astra doing that level of generalization now, or is there still a gap?” I think he’s talking more about the frontier models. I’ve seen a few demos with GPT-6 doing some robotics things. What are your thoughts on that?

Steven Palma

I think what these videos are missing is this: next time I see a video like that on social media, I’m going to ask the author to do the same application but with an ice cube, and then see what the difference is. Because you always have this 10x speed-up, with models that are actually very capable. We’re always very impressed: hey, they actually managed to do the task. But it takes a lot of time. If your application can handle that duration, then that’s okay, you’re good, you’re in the money. You can go and deploy these models as long as you have the budget for it.

But in my experience, robotics applications tend to be more latency-sensitive. So next time you see one of these videos, just ask the guy to put some ice cubes into the bowl, and then see how the ice melts over the duration of the video.

Pepijn Kooijmans

We actually tried it out in the office with Astra and other models as well. It is very impressive to see the generalization it does. They approach it in a different way, because they already have this huge model and they add this capability on top of it, all in the cloud, et cetera. Whereas most of the VLAs we’ve seen over the last year start from the requirements: they need to be small, they need to be able to run locally.

And let’s not forget, these VLAs are also based on off-the-shelf VLMs. If you reason through it that way, they probably trained on robotics data, but I don’t think there’s that much magic sauce to it, other than of course that they have a lot of data in other domains as well. So maybe it just generalizes a bit, and it’s just scale. But I don’t see why VLAs themselves or world action models wouldn’t be able to get there.

Especially since we already have these robotics systems or models that use a System 1 and a System 2, where the high-level reasoner can already be Astra, because the planning part, I think, is in general already solved. So then it’s mostly about having a very steerable, fast action model.

Steven Palma

Yeah, and following the same reasoning: maybe I gave the impression before that latency is the only thing we need to solve. But let’s say you put Astra into your robot and your model is fast enough. The way these models currently move the robot is that they come up with their own inverse kinematics script. They send the points, they figure out the 3D space of the task and the scene. It’s very funny, because now you have this AI model resorting back to classical robotics to solve the task at hand.

So sure, you can do it, in the sense that it is now zero-shot, because the AI, in this case the LLM, figured out the script the robot needs to execute the task. But to a certain extent you’re losing the flexibility part, the adaptability part. If you change the object, or maybe you change the scene, the script the LLM wrote before doesn’t really work anymore. And then it’s the LLM figuring out why. Actually, it would be very interesting, because now the LLM figures out that classical robotics is not enough, and then maybe it starts using a VLA.

Yacine Mahdid

It trains something on the side and just connects it. I think it’s using LeRobot.

Steven Palma

Exactly. So we have come full circle. But as Pepijn said, at least VLAs and world models have tackled all the challenges from the very root, to the point that they natively spit out robotics actions, instead of a script that you then go and execute.

00:41:43 – Frontier Models as Coding Machines

Yacine Mahdid

That’s one thing that’s very easy to forget, especially when you’re benchmarking a smaller model against these gigantic Claude API things: it’s almost not an LLM anymore. It’s a piece of software. There’s an LLM in there steering the stuff and spitting out code, but it’s literally a coding machine. So if your application has a coding component, in the sense that there’s a solution that can be found in code, they can work. It’s literally that.

And classical robotics was not about training a model, right? It was about figuring out deterministically what we can do, and there’s a bunch of handwritten algorithms that you can jam in there. That’s still what’s happening in some of the production stuff. So if you have a model that can spit out this code and jam it into your application, it can work. At first it will be slow, because it has to figure things out, but then it could be fast, because the code is going to be in there.

Pepijn Kooijmans

Yeah, and I also want to mention that of course Astra can also output actions directly. But it’s a bit funny to me. When I first saw it, I was really reminded of the first papers from what Google was doing with Everyday Robots. I think you mentioned it earlier as well, RT-1, Robotics Transformer 1, et cetera, where they basically did this: they tokenized actions and then trained an LLM on them directly.

RT-1’s architecture: images and a language instruction go through a Transformer that outputs discretized, tokenized actions, the approach Pepijn says Astra echoes. (source)

It’s funny, because we evolved from that architecture into adding a flow matching head and doing all these things. When you scale it up, it now works better, and Astra can do these things by directly outputting actions too. But I don’t see why you wouldn’t take in all the efficiency gains we’ve figured out in the meantime by having these VLA or world action model architectures.

Steven Palma

Yeah. And to the point of what we were saying before, where the model comes out with code that’s ready to execute, and the first time it figures it out it’s going to be slow but after that it’s going to work: essentially you’re using this AI to do system identification of your plant, of your model. Which is fair, because doing that system identification in the past has been very challenging, especially when you want to do it deterministically and need to consider all the friction parameters and all the kinematics and so on.

So essentially what we have is this AI model doing the system identification, which I find very interesting, because I believe models themselves can be very well used for the identification of the system. They’re essentially machines that are very good at solving differential equations in general. But then you go back to a VLA type of architecture, in which you have a backbone doing all the reasoning behind the system identification stuff, and then the action head producing the actions.

And going back to what Pepijn said, which I think was a very subtle point: in System 2 and System 1 architectures, your System 2 can be very, very capable and very, very big, and you don’t mind the latency that much. Our expectation is that as VLMs, these AI models, continue to get better and better, they’re going to be very good at orchestrating and leading this little System 1 model that might be running locally on the robot, the one that’s actually very fast and can interpolate all the different points. But System 2 can really stay as the organizer, the planner, the big mind telling the other one what to do.

00:45:45 – Toward a Robotics Harness

Yacine Mahdid

As long as you can feed the information in a way that’s optimal for it, I think it can work well. Do you have any plans, maybe not right now but in the future, to facilitate this, some kind of robotics harness in some shape or form? Because it is a viable way to do some of these things.

I’m working a lot with time series, and what I’ve realized is that if you just shove raw time series into these models, it doesn’t make any sense. They’re just very, very confused. But if you do a lot of the pipelining work beforehand to simplify the time series, to juice out all the main information for your application, and then give it to an LLM zero-shot, it can do a lot. It can do almost as well as, or sometimes better than, a model trained purely on the time series itself.

In one project it took me six months just to get the pipeline into a shape that would be useful. But then I have a lot more flexibility, because I now have this LLM, and with this harness I can do a bunch of things. I can have it use some of the tooling I have to go and start understanding the time series. I can also have it explain what the heck it’s doing, which is a bit harder with a smaller model; I have to do a whole bunch of gymnastics to get that. You get that natural-language flexibility afterward, which is pretty interesting. So, any plans along those lines? I don’t even know if robotics harnessing is actually a thing.

Pepijn Kooijmans

Yeah, I think that’s something we’ve been working on quite a lot over the last couple of months. It’s what we mentioned earlier: more steerable VLAs, or world action models, or models that control robots. Because of course, if you have this general planner that’s already very good and off the shelf, then the task for us in robotics is really to have a model that listens very well to instructions. So we’re adding a bunch of language support, so you can train with diverse instructions, long and short horizons.

We were also experimenting with having planners such as Astra directly control a robot, just so people can experience where this boundary might be, because I see it probably going up and down in the coming months as well. If Astra becomes more capable, you might want to give it a bit more freedom in steering the robot. And if the VLA then becomes better, you might want to shift that boundary up a bit.

And finally, we’re also already building this kind of harness a bit, where you can have System 1 and System 2 together. One could be an off-the-shelf VLM, and then you can basically define the boundary and have it instruct the low-level policy, or maybe even use multiple policies. You can imagine a VLA for navigation and then one specifically for manipulation, et cetera. Probably in the next release we will have a lot of exciting new tools for that. I’m also very excited to have people try it out, because I think it opens up a lot of possibilities, especially because you can lift on all the capabilities that current

...LLMs already can do, right? Such as general planning, et cetera.

00:49:11 – Platform, Library and Three Principles

Yacine Mahdid

That makes a lot of sense. The more I look at LeRobot as a library, the more it seems almost like a platform, an interface between all of these different parties. So what’s the general plan here? Is it to be a platform where people can plug and play and start to integrate their hardware, or just to be a very, very good library?

Steven Palma

I think both, right? I think those two things hold hands to a certain extent. I believe that if we create a very good library that has all the necessary tools for you to experiment with, that will not only let people use it more, it will also let people build on top of it. Which is actually the case today. We even have startups that came out after experimenting with LeRobot, and maybe are even based on LeRobot at different stages: the dataset part, maybe the policy part, maybe the benchmarking part.

So I believe it fundamentally starts by building this reliable library that follows the state of the art of the field. After that, people will naturally come into the project and start building things organically on top of it.

Which actually takes me to the three principles of LeRobot, which we haven’t mentioned so far, so I’m just going to shoot them out there. The first one is state of the art, or being contemporary. That’s all the things we’ve been talking about so far: System 1, System 2, using another model, maybe in-context learning, reinforcement learning, imitation learning. All these innovations in the space, we want to have them in the project, because this is what people want to experiment with today. So that’s the state of the art, the part where we translate research into something people can play with and experiment with.

The second part is accessibility. As we’ve been saying all this time, it’s very important for us to make these things as easy to use as possible. Not only through being open source, but also through cheap hardware, so that from the moment you decide to try it out to the moment you’re actually running, it takes you something like 10 minutes.

And the third part, which we’ve also already touched on but which is important to solidify, is community. This is not a project that we own and then just decide what to do with. We tend to listen a lot to what the field is doing, what the community is experimenting with, the latest videos or research out there, because that shapes what the project looks like as well.

So, tying the loop with the original question: all these three things help us have a very good library that people will organically build on top of.

Yacine Mahdid

Right. And the LLMs will also build on top of it, right? Because if they can interact well with it, it’s going to become the tool of choice for them too. So that’s pretty cool.

00:52:29 – What Porting a VLA Takes

Yacine Mahdid

I think it was you, Pepijn, who did the port of π0 and π0.5 to LeRobot, right?

Pepijn Kooijmans

π0.5, yeah. π0 we did together with the Physical Intelligence team, when they released it, on day zero. And then I did π0.5 later.

Yacine Mahdid

This is super interesting, because to make this whole thing work you have the hardware complexity and all that, but then you also have all the ports you need to do. And if there’s another version, you still have to follow it. What does that porting work look like for a VLA? What was the process when you were doing it?

Pepijn Kooijmans

In this case it was a little bit different, because they had already released a PyTorch version, I think. But what usually happens is that either the company or us, if we’re very excited about something new, reach out to each other. You basically state the intention: we want to have it in LeRobot, or they might want to have it in LeRobot. Luckily they often already have some codebase. It depends heavily on whether it’s a very researchy codebase, hacked together, or already a bit cleaner in terms of code.

Now the process is a lot simpler, I think. We go through the model and first try to figure out how it can fit into the interface we have. I think it’s a very good benefit that over the years we’ve built quite a nice interface that can suit a lot of models. Then it’s mostly about porting the modeling file itself, figuring out where it interfaces with things such as the preprocessors we have, the normalization layer we have in LeRobot, the dataset, et cetera. And then just a bunch of testing.

We can check the hash on loading the model and make sure all the tensors are the same, that the input and output are the same. Then we often also try to run it on a benchmark, such as LIBERO or a harder one, and run it on the real robot. So that’s the process in general.

And it’s actually quite hard nowadays, I would say, to figure out how much better a policy is, because we got so many policies over the last year, or the last couple of months. It’s quite hard to compare them aside from just reading the paper and speaking with the authors. LLMs you can try out: anyone can download a model from Hugging Face or get a ChatGPT subscription. But for robotics it’s super hard. You have to have the robot, you have to run it, and it’s not as easy to compare these 10 models and get a good feeling of what’s good and what’s not.

Steven Palma

And even after we successfully port the policy or the model to the codebase, it doesn’t really stop there. Just to keep hammering on the community part: when people go and try it out, they might find improvements to it. Then we had to decide whether we wanted to stay truthful to the implementation in the paper, or of the original authors, or have in LeRobot an implementation that includes all the little optimizations that came out of the community when they gave it a try and benchmarked it.

Ultimately, we asked the community: what do you prefer? The policy as it is in the paper, or with these little optimizations all around? People seem to lean a bit more toward the optimized version, because it condenses the learnings from all the experimentation around it, which I believe no single lab or single company can do. So that’s really, really important to the project: once we migrate a policy, people still have a landing point to contribute back based on their experimentation and their results.

Yacine Mahdid

And once the port is done, when the lab or company makes another version, I think they’re more incentivized to contribute back instead of doing this work again and having to redo the port, especially if they’re actually interested in bringing it to the community in the first place. If they want to keep it closed source and don’t care about that, then you don’t have to do this port work anyway. So yeah, this is cool.

00:57:22 – You Can’t Vibe Test a Robot

Steven Palma

Also, just a side comment on what Pepijn said right before, because I think it’s very interesting. LLMs you can vibe test, right? But vibe testing a robot is an entirely different thing. There’s just no easy way today, even if you already have the hardware. Let’s say you have what Pepijn has in the background, the OpenArm arms over there. And you say, okay, today a new frontier lab released a model, let me just try it out. It’s super hard. There’s no way you just change the model and everything works.

This is a bit what we try to do in LeRobot, but then we need to follow certain contracts, like the policy being implemented the way we expect, so that you can quickly switch from one model to another. But if tomorrow there’s a new policy with a new architecture, a new breakthrough and so on, it’s so hard for people to just vibe test it and see how much better it is compared to the one they already had.

00:58:26 – Survival of the Fittest Policies

Yacine Mahdid

And there’s so much being tested out on the robotics side at the moment. Do you guys have any thoughts or biases toward some policies versus others? Are there some that show much more promise, or that are maybe underrated? I’d really like your thoughts here, because you’ve literally tested a lot more than the average robotics person.

Pepijn Kooijmans

Yeah, last week actually I tested some new policies. While it is very hard, I like to run the robot much more often, and, like we were just talking about, lower the bar so you can try a new VLA every 5 minutes.

I do think the direction of world action models is very interesting. I’ve seen some intuitively very nice results with FLUX, for example, finetuning that, or some other policies from Robbyant. Also in-context learning, so prompting not only with text but maybe with videos. I think that’s very interesting, and I’m quite excited for that as well.

So we’re going to see in the end. I think we need a lot more rollouts, but those directions are definitely interesting. And the models I tried are probably a bit more data efficient and a bit better at generalizing to different objects, understanding the scene a bit better.

Steven Palma

Yeah, and here I have two quick things to say. On the policy or model side, we try to see everything in the field as survival of the fittest. If people have it accessible, they’ll just give it a try, experiment with it, and naturally that will drive more interest toward the ones that actually work very well, based on the different constraints people might have.

This is the importance of open source and projects like LeRobot: it lets you pick the flavor of model you want, and based on that you decide which one worked best. Then that resonates with the rest of the community. You go out and say, hey, I tried the VLA, or maybe I tried world models, or maybe I tried in-context learning, and that worked way better than anything I’ve seen before. And that gets the snowball rolling over time.

So I think it’s important to have that survival of the fittest among policies, because from the LeRobot perspective, it’s not our mission to decide where the field should move. It’s more about the community having its own criteria and contributing that back into the field.

The second thing I wanted to say is even broader. As you said, Yacine, there are so many things happening right now that it’s sometimes hard to decide where we’re going to spend our time, where we’re going to put our efforts, what’s the new thing we’re going to implement. For that, we go back again to the community: to the research labs, to industry, even to the things we see on social media. They help us decide where we can spend our time best, and where the users of the robots will benefit the most.

Yacine Mahdid

Yeah. But I really feel like the fact that you guys are at the nexus of all these different policies, and that it’s literally your job to make this thing better for everybody, has an impact on how the optimal solution gets shaped and allowed to emerge. If you look at the past with the Transformer, that was the bet: the Transformer is very exciting, it seems worth maintaining, and the library should evolve beyond that.

There was an organization that decided, let’s make this thing much better and much easier to run for everybody, and let’s centralize it. I might be wrong, but I think that helped shape the direction we went in, and a lot of people were able to contribute to the findings. It’s not just some closed lab that thought about everything on its own. It’s a community effort, and it’s still people pushing the frontier. The fact that the tools are so easy to use now, like with LLMs, plays a role in the direction we’re going.

That’s my two cents. If you decided, “You know what, I don’t care about this specific type of model,” and you didn’t port it, there would be fewer people using that stuff. So the fact that you’re not biased there is a really good move.

01:03:19 – A Little Book Everyone Can Read

Yacine Mahdid

I wanted to talk a bit about the datasets. I thought I saw 77,000 LeRobot datasets, but it’s actually 88,000, so it’s growing. All of these datasets getting uploaded come from different configurations, different hardware, different ways of collecting the information, different sensors, and it all gets structured in the LeRobot dataset format.

Do you think this dataset that’s becoming the hub of robotics is enabling some sort of generalization? Not right now, but in the future. With the internet, the data is just over there. We could grab it and get started building foundation models for LLMs. With robotics you don’t have that. Yes, you could scrape the internet for videos and stuff, but it’s not the same thing, because you don’t have the actions. But now you’re starting to structure this, and it’s actually growing. So what are your thoughts on this front, in, I don’t know, five years, when you have a hundred times that amount of data or more?

Steven Palma

I might be biased, but I think we are already making a dent with these datasets being available for every frontier lab and every hobbyist out there. We’re talking here about 88,000 datasets on the Hub, which I think is even more than 200 terabytes of data. And that’s only the LeRobot format. On the Hub we also have other robotics-related data in different raw capture formats, which are very rich in terms of features, and which haven’t been converted into LeRobot.

But the point is, we’re definitely contributing to this library of physical AI data out there. Whether it’s going to be the biggest one and the most important one, I’m not sure. I’m not here to tell that. But I can confidently say that these 200 terabytes will continue to grow over the next five years, and by then we’ll definitely have our place in whatever pre-training is happening in the robotics space.

So it’s like having at least a little book in the big library. But it’s important, because it’s a book that everyone can read. Everyone can go and use the dataset, analyze it, learn from it. I believe that’s one of the most important parts of open source in the robotics space.

Steven’s image for the Hub’s robot data: one little book in a big library, but one that everyone can read.

The reason open-source LLMs keep staying so close to closed LLMs is that they had data accessibility, and they could eventually distill the models. Neither of those is really true for robotics foundation models. Access to the data can essentially be privatized: you run your own experiments and collect an immense number of hours of data per day, which you’re not putting out there on the internet.

And second, to distill a robotics foundation model, you’d need to run your robot in 1,000 scenarios for 24 hours a day. From a scalability perspective, that’s of course not as financially viable as it is for LLMs, where you can just spawn 1,000 or 100,000 agents doing simultaneous queries on different topics. Imagine doing that for a robot. You have a robot picking up cubes today, then tomorrow robots pouring water. It’s just going to take you forever.

So having these datasets available on the Hugging Face Hub is definitely a good step toward letting people build and learn. As I said, it’s a little book in the library, but it’s a book that everyone can read.

01:07:57 – No Bad Data, Only Bad Labels

Yacine Mahdid

Well said, man. That was good. Also on the data side, I saw that you both worked on the SmolVLA policy, and I think you curated and created a whole bunch of demonstrations for the robot. From your perspective, what makes good demonstration data versus bad? What are the characteristics?

Pepijn Kooijmans

For me that has actually changed a bit over time. Take the folding experiment. It’s one task, but we still tried to really generalize it to different t-shirts, tables, environments, heights of the robot and so on. For one task, the way we trained it with imitation learning, it was very important to have clean data of the action itself. Because you train with chunks, you want the signal in a frame to tell you where it is in the task.

So you want a clean demonstration of every step of the folding. Not exactly the same way each time, but in a way where, for example, the beginning doesn’t look like another step in the episode. No real ambiguity there, basically. Practically, that means you still want some order in how you record the demonstrations: a kind of controlled variety in the t-shirts and so on.

However, nowadays, with these more steerable ways of training models, I feel like almost no data is bad data, especially with world action models. If it’s labeled correctly and the model is actually trained on “this is a bad demonstration,” like what you see Pi doing now, where you give demonstrations a speed label, a quality label and a very correct text label of what the demonstration is, then I feel like you can’t really have bad data. Only badly labeled data.

Because in the end, it’s all about connecting the text embeddings, the vision embeddings and the action embeddings. It is bad data, of course, if you have one task and a bunch of different ways of executing the action, because then it’s super hard to find any connection between those. But if everything’s labeled correctly, you can imagine the model actually knows that if you fold it like this, it’s a bit less neat than if you fold it with five stars. So nowadays I feel it’s more about labeling correctly and making sure that, overall, these embeddings are very clear.

Yacine Mahdid

And with respect to the FineART paper that you’re on, the fact that there are sub-labels and little subtasks in there makes having one bad, garbage trajectory almost not a real thing. You could always have the subtasks that were done well, and then maybe at the end it was messed up. So it might primarily be a labeling thing.

The FineART annotation schema: a label for the whole demonstration plus time-segmented subtask labels, so the good steps of a failed episode still count. (source)

Pepijn Kooijmans

Especially with world action models, you can train on negative examples. If something falls on the ground, you can say, “Throw the shirt on the ground,” and still learn from that. And I think in general it’s a trend. Initially we trained models, and a lot of models in LeRobot still are, on just action supervision. You already have the VLM there and you try to preserve it, but it’s not actively conditioning on internet examples.

Over time we’ve seen with VLAs as well that, besides robot data, we also try to keep training on text and image data. Now with world action models it’s even richer, where you try to predict the future directly. All of this gives models a more natural way of learning from the data that’s available. Which also means we can now learn from data that’s not specifically robotics, because the model is shaped more naturally: it can learn from general videos, human videos, text data, or a mix of all of these. That’s the intuitive feeling I have about robot data, and data in general.

Steven Palma

I can add a couple of points. If I were to summarize the feeling here: the better your dataset is annotated, and of course the better your policy architecture, the more entropy you can take on in the dataset itself. What I mean is, as Pepijn said, say the robot dropped the milk. If that’s still annotated, it’s okay. It’s still a valid episode, because the model will eventually learn that this is an episode where the milk dropped, instead of you filtering out the entire thing. So what I’m trying to say is that the better your dataset is annotated, and of course the

...the more capable your policy architecture is, the more entropy you’re able to take at the data level.

The second thing is that although we can go very granular on what a good dataset is and what it looks like, I still think there are some basic rules we can all agree are part of what makes a good dataset. On my side, for example, I’d say the more raw sensor modalities you can acquire at collection time, the richer that dataset will be, and that makes it a better dataset over time. Today we use dominantly image-based sensor modalities, but in the future we might need more. So if you already took care at the collection level of having different modalities in your dataset, that allows better experimentation in the future.

You want to avoid the scenario where you spend thousands of hours and maybe a big part of your budget collecting data, and then three months later a researcher comes to you saying, “Hey, wait, your dataset didn’t have this modality.” And you’re like, “Oh no, I need to do it again.” You don’t want to do that. So for example, a good dataset has different modalities.

Then there are basic things like: you don’t want your camera all blurry, you want it sharp. All these very basic things continue to be true. Sure, if we go more granular about what an episode can be and how specific parts of the dataset can be useful, then we can talk about subtask annotation and all the different things we’ve been mentioning so far. But on the broad side, there are still multiple things that make a good dataset, in my opinion.

Yacine Mahdid

Yeah, yeah. In the case of digital learning, you change your environment and let it roll, and if you’re strapped for time you just use more GPUs. But if you have to jam a sensor in there, first you have to get the right one, put it there, do a bunch of tests, and then rerun the whole thing.

And it’s hard to see what will be good for a given problem. In some cases it’s surprising: the vision component is super good at this, but for this specific subtask vision is not good, and it’s this other sensor, more of a time series, that is better. You don’t know, you just have to collect. But it’s also physical, so you don’t have the luxury of jamming all the sensors in there. So, pretty cool.

01:16:33 – Why Open Robotics Makes Us Safer

Yacine Mahdid

I had a whole bunch of other questions, especially around one of your essays, Steven, titled “Why I Believe Open Robotics Makes Us Safer, Not Less Safe.” I think it’s very topical, with LLMs beginning to get better and all these scares about capabilities we weren’t foreseeing. Can you tell us a bit about your core argument? Because I actually agree with this, 100%, but I want you to explain your rationale.

Steven Palma’s essay “Why I Believe Open Robotics Makes Us Safer, Not Less Safe,” published on Hugging Face Spaces. (source)

Steven Palma

Yeah, absolutely. I think this is a very important question, because we’ve been advocating all this time for open source. My experience is that when I go and explain the robot and all the exciting innovations in the physical AI space, people who aren’t very familiar first think of a Terminator-like scenario. Which I think is fair, because those are tangible things, so we need to tackle them in a sensible manner.

A lot of the conversation is: “Hey, wait, if you make this technology available for everyone, that will make it easier for a bad-intentioned character to use it for bad.” So in this article, I’m trying to explain why I believe the open-source ecosystem gives us a better chance of preventing that Terminator-like scenario than closed source.

I’m not saying that transparency, and by transparency I mean open source, eliminates the risk. The risk is still there. It’s that transparency gives us visibility into the problem, and visibility is usually where safety can start, because you have more people involved. Researchers, university professors, policymakers, manufacturers, engineers: all these people can decide collectively how to shape the technology together.

So essentially what I’m trying to answer is which ecosystem gives us the better chance of a bright future, a closed one or an open one. I go through different arguments for why I believe an open one is the way to go. It all comes down to transparency, because openness in general is how we as a society learn to understand, audit and eventually manage these powerful technologies. That’s the entire argument.

The different steps I go into are specific. We don’t want to end up in a scenario with one centralized power. We don’t want a scenario where we restrict all research just for the sake of protecting against what it could be. There are better places, later down in that process, where we can restrict.

I actually encourage everyone watching this video to give it a read, because it’s very important for us to be able to answer when someone asks, “Wait, you’re working on open source, so that means everyone can just use this technology to do something bad?” And it’s not. As I said at the beginning, transparency doesn’t eliminate the risk, but it makes it more present, so that everyone can talk about it and shape how it works.

Yacine Mahdid

Yeah, 100% agree. Just to take the example of frontier LLMs: if there were no open source, no open-weight models, and the labs still managed to get this thing started and we had no clue how they work, I think the situation would be much scarier than it is now. Right now we know how they run, and we can test a bunch of stuff. We can test the behavior of the closed-source models, yes, but also the open models, and when we see their internals we know what their limits really are. So it helps us prepare for whatever the hell is going on.

For example, we all know these LLMs are now coding machines; that’s kind of a known thing. So if you have poor sandboxing, of course you’re going to get bad things happening, and they’re going to start finding ways of getting out, because they see code, and it’s not just an LLM anymore, it’s all the harness around it. I think it’s one of the Ilya tweets that says you should just fix the sandbox, that’s priority number one.

So generally speaking, it also helps the whole ecosystem understand what should be prioritized to have a better defense. If we didn’t know anything about that, and people just started seeing waves of weird things happening on their stuff without understanding that these things are scouring the code and understanding it at a deep level, then it’s another story completely.

01:22:26 – Constrain the Pipe, Not the Research

Steven Palma

Yeah, absolutely. Taking it from the LeRobot perspective: LeRobot is the open-source project, and it gives you the tools to experiment with. Let’s say someone comes tomorrow and tells us, “Hey, I think you should stop that, because it will allow people to understand how to use robots in a bad manner,” which is the actual scenario we’re trying to avoid.

What I believe, at least in the case of LeRobot, is that if we restrict that type of information, that type of learning, those types of tools, it mostly puts at a disadvantage the people trying to study the problem, the people who will be capable of identifying this risk and understanding what’s going on. All these students, researchers, educators are learning; this is the next generation learning about all these things, so that we’re better prepared when we need to ask questions as a community, as a society. If we restrict research in general, we’re sacrificing all these people in the hope that it will stop ill-intentioned people.

Let’s take a step back. I’m sorry if I’m overextending here, but I think this topic is super important.

Yacine Mahdid

No, it’s good. Keep at it.

Steven Palma

Let’s say I’m an engineer and I want to make a bad robot, a robot that does harm. In that process I’m going to do different things. I’m going to need the research, the hardware, the electronics, I’m going to need to find the people, and so on. You can see me going from point 0 to point 1 as a pipe. At the very beginning of that pipe is just the research and all the technology that enables it, but there are different steps in the pipe.

If we constrain the first step, the research, it’s a very inefficient way of doing it, because, as I said, we’re sacrificing all the students, professors, researchers and policymakers, all these characters who can actually contribute something positive to the field. We’re throwing them away by saying, “No, we need to stop it over here.”

So the argument is that in that pipeline from 0 to 1, there are better places where you can constrain a bad-intentioned actor or a company getting to 1. In the blog post, I said hardware, for example, is today one of those bottlenecks that’s way easier to control, and controlling it doesn’t limit access to learning and education for all the other people who might contribute positively to the field.

Yacine Mahdid

Yeah, true. If you want to do harm at a large scale, you have to find the supplier, do the back and forth, send the PO, there are delays, and the supplier needs to understand what the heck they’re doing. It’s pretty complicated just to make regular stuff. That’s plain old manufacturing, getting all your parts correct and so on.

Steven Palma

Which is fundamentally different from how software works today, right?

Yacine Mahdid

It’s not the same thing at all. And for controlled goods it’s even worse than that: you have to jump through all the hoops. It’s not that it’s a non-issue. I just find it’s not the right part of the problem to look at. It’s literally the wrong part.

It’s also much easier to not constrain the research, and instead constrain the access to whatever material would be problematic, and regulate that or monitor it very precisely. Because how are you even going to do it otherwise? If people are very interested in VLAs, they’re going to figure it out. They’re going to find a way to get that knowledge and understand it. They’re going to tinker with it. They’re going to move to a different region that doesn’t have the same regulation. It’s a wasteful exercise for not much.

Steven Palma

Yeah. A good example off the top of my head would be Wi-Fi. Imagine it’s the early days of Wi-Fi, and we say we have a big problem with Wi-Fi and we want to stop bad actors. So do we stop all research about it? Do we put behind closed doors how Wi-Fi works, how we actually transmit data around, and all the things we need to benefit from it, just to prevent someone from uploading a video that might not be appropriate? That would be very expensive, and it’s not the best place to put all our efforts.

First, because the technology has a lot of upsides, it’s very beneficial. But also because it allows people to learn about it and understand how it works. So with Wi-Fi, we built the bottleneck later in the pipeline. We go to the application layer, we identify the potentially malicious video and say, okay, this is banned from our platform. That’s how we stop the people with bad intentions. We didn’t say, hey, we have to stop Wi-Fi research, or maybe continue it but behind closed doors. We didn’t do that. So I think it’s the same reasoning for physical AI.

Yacine Mahdid

Yeah. If you think about it, the internet is a crazy thing, absolutely insane. And we still have it, and I think it’s a net positive in general.

01:28:07 – Is There a Scaling Law for Robots?

Yacine Mahdid

I showed you the paper on scaling laws, literally today. But that’s one chunk of it, something like 200 million to 8B parameters. Do you think we’re close to finding the scaling law for the later part of growing these models in robotics?

Pepijn Kooijmans

Yeah, I think we’ve seen quite some nice papers and also posts from companies, mostly over the last year, about scaling laws, especially data scaling laws. I think they’re definitely there. But aside from just scaling data, there are still so many unanswered questions. What kind of mix do we need to scale in? Do you want more egocentric, more teleop? What’s most efficient there is probably the right question, because egocentric is much cheaper to collect. So is it more efficient to do 90% egocentric and only 10% very good, high-quality real use cases, for example? Or is it a different mixture?

So we’ve seen some good indications of these scaling laws. It’s really a question of what the mix is that allows the scaling laws to still be valid.

Aside from that, one interesting observation, which is more about capability: let’s imagine that we have all the data we need, and hopefully we’ll get there soon. Would a 10x effort into scaling a model be worth the output you get from a model that’s a bit better? I think that question is quite interesting to explore for robotics. It’s also quite similar to small local models versus big cloud models, because I can imagine that at one point it’s hopefully enough.

We’re not there yet, these models need to become way better. But I can imagine that if you have a robot cleaning your house, from a motion or action point of view, it doesn’t make sense for it to run as fast as Usain Bolt. Maybe for manipulation that’s a bad example, but if it needs to clean your house, it maybe doesn’t need to do very dexterous things that only 1 in 100 humans can do. So I feel like the added benefit at some point will be smaller in robotics than it is with LLMs, where you can have a factory of researchers, like Anthropic says.

So I think we’ve seen some clear indication of scaling laws, but it’s still a question of what the mixture will be, and how big we want to scale, or need to scale.

01:31:24 – Many Parts, Many Scaling Laws

Yacine Mahdid

Yeah. I’m trying to make the comparison to LLMs, or even raw VLMs, and just call it a day, but I don’t think that actually makes much sense. It’s almost as if there are multiple scaling laws for different components of the system. Take the cleaning robot: doing the actual cleaning is one thing. But then there’s maybe a planning component that could be super slow. It could take the whole night, and it’s still feasible.

Same thing from a manufacturing line perspective. Okay, the defect rate here is 2%, with this latency, and it takes that much time. That part can have its own scaling law, its own thing running. But then you can have the orchestrator that gets all this information, and it’s slow as hell. It takes a month or whatever. But then it finds how to drop the defects by 1%, or make it twice as fast. That’s something else. I get the feeling that having just one thing isn’t necessarily it, though I have absolutely nothing to base that on. What’s your thought on that?

Pepijn Kooijmans

Having one single scaling law? No. With everything in robotics, the first thing is that everything needs to be good enough. And there are more components. You can argue that LLMs also need the computer and so on, but with robotics you have the hardware, you need the high-level reasoner, you need something to execute the motion, and you potentially have human interaction. So you just have so many more systems.

And if you multiply the reliability of each system, the whole thing becomes less and less reliable the more parts you add. Which means robotics is just harder, because you have to do everything right, and there are a lot more things that need to be correct. So scaling data is one, but I think there are a lot more scaling factors, if you can call them that, that need to be done correctly.

01:33:53 – How Do You Test a Generalist?

Yacine Mahdid

Steven?

Steven Palma

One interesting thing goes back to what Pepijn said: if my model is 10x better than the other one, what difference does that really make? For me, it would be very exciting to see research on the minimum number of scenarios I need to test my robot in to guarantee a certain level of generality for a specific task.

Because as models get better and better, let’s say my model is 10x better than yesterday. What does that mean? Does it mean I can pick up 10 different types of cups, or pour 10 different types of liquids? And how am I going to test all that? Simulation plays a big role here, evaluating infinite scenarios, or a thousand scenarios, with different backgrounds, different cups, different liquids and so on. But where do we stop? If tomorrow the model is 100x better, does that mean I’ll need to spawn 100,000x more simulation scenarios to guarantee that my model actually is that?

So from my side, as I said at the beginning, it would be very interesting to know the minimum number of scenarios I can test, maybe in simulation, that’s enough to claim that my model is general enough for these types of scenarios, for these types of tasks.

Yacine Mahdid

Yeah, I think this is super important, because I have a physical AI project starting soon. Long story short, they have to create defects in order to assess the quality of something. They create the defects with their client when they deploy the system, and then they train on these defects. But that costs 10K every time, for every batch of 500 data points. So they’re always looking at whether they can do just two batches and be done. And the client wants to do no batches at all, because it’s costing them product they could literally be selling. So that has a very big impact when you go to the real world, for real.

Steven Palma

Yeah. The example I usually like to mention: let’s say you wake up tomorrow, Yacine, and you step out your door, and there’s an auditor who tells you, hey Yacine, I’m going to follow you around to understand whether you’re a generalist as a human. I’m going to evaluate you throughout your day, and at the end of the day I’ll tell you whether you’re general. How many activities would you have to do to be considered general as a human? It’s a little bit hard.

So how can we translate that to an actual robot? Because the robot will eventually need to be benchmarked to that extent. And that’s maybe part of what Pepijn wanted to say before: if I have a robot in my home, do I really need it to know how to do a backflip, or how to climb Everest? Maybe not. So getting an intuition about how good these models are and what they’re good for, I think that’s a very exciting part of research I’d be very excited to see. A kind of scaling law in that regard.

01:37:21 – Bootstrapping on a Vision-Language Model

Yacine Mahdid

Yeah. I was reading a bunch of papers, and I think it was Physical Intelligence that did the knowledge insulation thing, to not mess up the VLM in this mix. And it got me thinking that we’re jumping through a bunch of hoops so we don’t mess up the thing we’re using to bootstrap everything.

A figure from Physical Intelligence’s Knowledge Insulating Vision-Language-Action Models paper, which trains the action part without degrading the pretrained VLM backbone. (source)

And it’s very useful to have a VLM that’s already pre-trained, because you’re going to give instructions in language to the thing anyway. And if you can have that interface, and the fact that it can create the action from it, it makes the labeling

…easier and all this stuff. So language does have a big component to it. But I’m wondering: do you think this backbone is almost a crutch we have at the moment, and at some point it will be completely popped off and replaced end-to-end? Or is it actually a very crucial component of this whole mix?

Pepijn Kooijmans

I think it’s probably quite a crucial component. Maybe if you had all the perfect data it wouldn’t be, but we don’t have the perfect data yet. The perfect data would be real use cases, real human interactions, fully end-to-end. But we’re not deploying these robots that much yet, so we don’t have this perfect data. Which means I think we do need to leverage what LLMs and VLMs can already do.

As I said earlier, we try to bridge all these domains so that, hopefully, the robotics action gets learned more efficiently, and we can already use the priors from vision and text. So in that regard, I think it’s very important, probably, to start with.

However, you can also reason that maybe now you can just distill this data from VLMs or LLMs and then natively train with robotics data, which means you can maybe ground the data a bit better. You can come up with not just text on random celebrities, but already enrich the robot data with LLMs and VLMs so it’s really about the robotics data, and really form these bridges that have a component in action, in vision and in language at the same time.

Steven Palma

My take on that is that successful systems cannot have a crush. They cannot be opinionated, they need to be pragmatic. What I’m trying to say is that most of the time, as it is in engineering, you’ve got to find your sweet spot. For a specific system, this architecture, or maybe the new one, or maybe the one we already thought was dead, might be a good fit. But it’s just part of the bigger system that encapsulates your application.

This is why I believe successful systems out there don’t really care that much whether you’re using one VLA or a stronger backbone and so on. You need to build your product, your solution, knowing that you might even replace those components as you go. But that will not ultimately determine whether your system is automating, whether it’s performing the task successfully, especially for the people or startups building vertical robotics. Ultimately you use whatever works best, and you go with it.

Yacine Mahdid

Yeah, you’re trying your best, right? I think in the π series, in one version it was two-step, in order to prevent the collapse of the VLM. But then in the other one, with knowledge distillation, you just do it in one shot, and it didn’t work. It seems like it would just continue in this direction, and the end goal would start to get optimized toward robotics and shine there.

What I’m trying to say is that even if you start with a backbone that is very general, at some point it’s fine to remove stuff that isn’t needed, and then you have something more streamlined. Not training the VLM based on what the action is doing makes a lot of sense. But maybe after some time we see another configuration, and if it just works better, I think the model will keep shifting in that direction.

01:41:47 – Are VLAs Dead?

Yacine Mahdid

We have another question about VLAs, by Utkarsh, who’s asking: “Are VLAs dead, with world action models being the next shiny new thing to work on? Is there actually a future with WAMs, considering they have a different approach in the domain of robot learning?” I think we touched a bit upon this, but what’s your general reaction?

Pepijn Kooijmans

I think it’s probably a bit more of a naming issue, to be honest. VLAs can be world action models, in my view, and world action models can be VLAs. They often have similar inputs. The question is just what you condition on.

I think VLAs are already more than just conditioned on action. Even with π0.5, you see that it’s also trained on just text examples, or vision and text pairs: bounding boxes, subtask prediction, these things. So in a way, we’re already conditioning on way more than just the action itself in VLAs.

With world action models, it feels to me like just more richness, because we’re explicitly, or in latent space, predicting the pixels. It’s just a step up from predicting bounding boxes and actions and a bunch of other things at the same time. Now we’re predicting exactly the future, basically, and the actions at the same time.

In terms of data, it’s a very natural progression, I would say, because it allows us to train on more data that’s non-robotic. You can probably hold a big discussion if you don’t listen to the other side, but I think it’s a very natural progression of just having more richness to train on.

Steven Palma

Yeah, world models are absolutely a very exciting branch to explore, and they might become the dominant player in the future. But to answer whether VLAs are dead or not, I think we should also ask: dead for whom? If you have your cookie factory, maybe a VLA is enough for your application, and you don’t need to deal with all the complexity, all the compute and all the research of a world model.

Maybe, you know what, even ACT will be enough. Maybe you don’t even need a VLA. Maybe I’m taking this conversation mostly from the engineering side, but as an engineer, you pick the one that fits your needs best.

Yacine Mahdid

It’s just whatever works, man. I’ve got a project where XGBoost is still the king. I have this whole fancy setup, I’m post-training this thing, and XGBoost is doing super well, and it’s interpretable. We can actually interpret a bunch of stuff in there.

I get this forward-looking view of what the next general thing is. But in the real world, it’s not always like that. Sometimes it’s: wait, why did you pick this very simple model? Because it works. It’s the best one. What do you want to do, give the worst experience to your client or your boss? No, man. You do the job and then you move on.

It’s fun to use something general because it gets you started faster. Even with classification, say, you can use a classification model zero-shot. But you don’t have to stop there. You can take the data and make something, and maybe another model will be much better.

01:45:32 – Data, Distribution and the ChatGPT Moment

Yacine Mahdid

We have another question, by SquaredCube: “Do you think data is the answer, or are we still waiting for our GPT?” There’s a lot of excitement for robotics these days. “I know a lot of people said that Dyna”, with the Dyna-2 run they did, a million-hour scaling law for world action models, “proved that data was the thing we were missing, but I don’t know if you think the same. Is there still room for a new paradigm, or are we just scaling from here?”

Dyna-2’s scaling-law figure for world-action models trained on up to a million hours of data. (source)

Steven Palma

Yeah, I think we’ve touched on that a little bit. My feeling is that data, although it’s the most obvious bottleneck today, is not the only one. There are still other breakthroughs that will enable robots to be in the wild.

To get to the point where we wake up, go to work, and seeing a robot in that process becomes normal, I believe robots need to become 10x better at what they do and how they do it today, but also 10x cheaper. And that is not only data. That comes with an entirely different set of challenges.

Pepijn Kooijmans

I would add the same. One part is, of course, the technical problem of making models that are good enough. Data is definitely one of the solutions for that, diverse enough data, at least a threshold we need to reach.

But aside from that, to me the GPT moment with LLMs, with ChatGPT or GPT-3, was more of a distribution thing as well. If you remember, before that you had the OpenAI Playground, but you didn’t really know what to ask or what it could do. And there wasn’t really this instruction interface.

I think those are two things we’re actually missing for robotics. One is what Steven is saying: having cheap robots that are distributed, so everyone has a robot in their home to try out these models. Otherwise the technology is there, but if nobody can feel it, it’s not a shared moment, I feel like.

The second thing is this instructability. Let’s say you have a robot in your home, the way we currently have it, where you can ask a robot or a VLA to do a short task. I don’t think that’s a GPT moment yet, because picking up a glass doesn’t feel like the GPT moment. So there’s also a lot of work to be done on how we have robots around us and how we instruct them, in manufacturing but also in the home.

Especially if you think about edge cases. One example that’s used often: I might have a favorite t-shirt that I don’t want machine washed, because I want it hand washed. How is that interaction going to go with my robot? Am I going to tell it to do the laundry, then send it a picture of this specific t-shirt and it needs to reason over it?

I think text alone is probably not sufficient there either. So what we’ve seen with in-context videos is very exciting for me as well: maybe show the robot an example once of how things are done in your home, and the robot can understand it, similar to how you might instruct another person who’s new in your home, or a new co-worker.

Steven Palma

I’ll follow up here, because the idea Pepijn just raised is super important: the ChatGPT moment arose because everyone could feel it. Of course there’s the distribution part. But there’s also the side where you go to a text box with zero expectations, you say, “Hey, write me a poem,” and it does it. And you think, “Wow, that’s something I wouldn’t be able to do.” That’s the ChatGPT moment: “Wow, these things are actually so capable.”

But with robots, the threshold starts way, way higher, because as soon as you see a robot with a human interface, like a human body, you automatically assume that the robot can already…

…do what the human can do. And anything less is actually a disappointment, not an excitement. Which is quite the contrary of when you interact with ChatGPT, right? Everything was an excitement, and there were only little things it could do to disappoint you.

So to get to that moment, I believe that of course we’re going to need to have robots around more often, but we’ll also need to see them doing things that make us think, wow. And those wows are not the things I do every day, because those things aren’t exciting for me. If I see a robot doing the washing machine, I say, well, you’d better, because that’s what you’re supposed to do.

01:52:03 – RL Post-Training for Robots

Yacine Mahdid

So I think the backflip component is very important. They should all be able to do a backflip straight out of the box. You get it in your home, you ask, and it starts with a backflip. Okay, there’s something there.

I think it’s the last one, by Arpan Dasgupta, asking something very interesting: how much is post-training, RL specifically, different in robotics than in language models? Is the sim-to-real gap the only differentiator? What do you think, Pepijn?

Pepijn Kooijmans

That’s a hard question actually, because there are multiple things in here. For locomotion, for teaching robots to walk and use their legs, the field uses a lot of sim currently. You train the robot on a variety of surfaces, and through RL in simulation it learns to walk. But it only has a few contact points, right? Just the feet.

If you talk about manipulation and using RL, I don’t think we’ve figured out the perfect post-training recipe there yet. We have HIL-SERL in the library, and we’re also working on integrating some even more promising RL algorithms. But manipulation specifically is a bit harder. First of all, the sim-to-real gap was, and maybe now it isn’t anymore, too big to bridge immediately, because you have to simulate contact forces, different objects on the table. If you look around your room, it’s probably already too messy to perfectly simulate.

So that leaves the real world. And doing RL on robots in the real world is very hard because, previously, maybe now it’s a bit different, we didn’t really have the priors of a VLM or VLA to at least get some decent motion going. So what you see, for example, when you run HIL-SERL on the SO-100, is that it starts from nothing, knowing nothing. It needs to grasp the cube, but it’s just going to start moving randomly. Luckily, if you have enough examples, combined with imitation learning, you can learn a simple task like picking a cube.

But it’s not as efficient or as easy as imitation learning yet, because you start from nothing. In imitation learning it’s a bit easier, because you already have these full examples and demonstrations. We haven’t really figured out a good RL post-training recipe yet.

But I feel it’s a very interesting direction, especially once the VLA can already do something, when it goes in the right direction but a little bit slowly. There’s definitely a big opportunity for RL to make the policy quicker, and make it a bit more accurate if you have specific insertion tasks. So we hope to work more on that soon, and we have a very good colleague working on it, so hopefully people can try it out as well.

Steven Palma

Maybe I’ll jump in here, because I think we haven’t done it so far, to encourage people to give the project a try. Check out LeRobot, get an SO-100 and start playing with robots. We don’t stop at imitation learning; we also have some reinforcement learning stacked into it, as Pepijn was saying. This is the place for people who are curious and want to experiment with the latest things in robotics. So definitely check it out: lerobot.com, or, well, it’s actually on GitHub, but yeah, check it out.

01:54:58 – What’s Next for LeRobot

Yacine Mahdid

Cool, fantastic. Then my last question is: what’s next for you guys and for LeRobot, and what are you excited about?

Steven Palma

Well, there are many things. We’re working on a lot of different features and projects for 0.7, which is going to be the next release. You can check it out in its own GitHub issue, where we lay down all of our ideas and objectives. As usual, it involves some work around datasets, some work around simulation, some work on the policies, and some work on inference time as well. So I expect a lot of improvements all around the project. And for me personally, I’m just excited to see what the community continues to build on top of the project.

LeRobot’s Hugging Face page, showing work built by the LeRobot community. (source)

Pepijn Kooijmans

Yeah, I very much agree. I’m very curious to see the community and also the field trying out different things and getting more insight into everything we’re trying as a field. Personally, we’re working a bit toward making sure people can also train bigger models, or models with more diverse data, natively in LeRobot.

So I really hope we can give people all the tools and all the data they might need to figure out the next steps in robotics, to experiment very broadly, but at the same time also be able to scale up experiments if they want to. I’m very excited about that. In general it’s a very exciting field, so if you’re in another field, feel free to check it out.

Yacine Mahdid

To join in. Yeah, like you mentioned, Steven, it’s a galaxy. So you might already technically be doing some of this stuff, and that’s your way in. Be comfortable with the fact that there are a lot of moving parts you might not understand, but they’re all connected together. It’s a very exciting time.

So thank you, Steven. Thank you, Pepijn, for coming. It was super cool to chat with you about all this.

Steven Palma

Yeah, thanks again, Yacine, for the invitation. And thanks everyone for tuning in and learning more about what’s going on in the robotics space, and in LeRobot as well.

Pepijn Kooijmans

Thanks.