LLMs were able to improve so quickly because there was a huge preexisting corpus of language data that it could be trained with available for (practically) free on the Internet. There is no corpus of physical data in the required format of comparable size (VLAs are not sufficient for general purpose robotics, so images are not enough!), and building one will have a non-trivial cost, and require a great deal of expensive policy coordination. So I think there is reason to believe we will not see the same rapid improvement due to scaling that we saw with LLMs.
The consumer and industrial potential for robotic AI is such that it's not hard to imagine billions being spent to develop massive data on physical/robotic movement. For robots that mimic human movement — the ones that easily capture our interest — the corpus could be something as simple and available as the massive amounts of video data being generated by humans daily.
The important claim that roboticists like Rodney Brooks have made is that video data is not enough - you also need tactile feedback. You can’t learn how to strike a match, or how to safely grip a slippery wet dish, just by looking at where your fingers are - you also have to *feel* what your fingers are feeling.
Having humans teleoperate robots while wearing force feedback gloves may be useful for generating training data, but even figuring out what pattern of sensors the gloves need hasn’t been done.
Yes, if we want to make robots that change diapers or give therapeutic massage, we may need that type of data. But industry, pre-AI, has been producing tactile dexterity in practice for decades. The argument was that the segment of robotic AI that theoretically requires a mass of tactile data was such that it would hold the entire project back. I don't think that's so. And we'll no doubt see huge advances in robotic AI on a commercial level in the coming decade even without the feedback that Brooks and others talk about. And I'll go farther and say that even the data gap will close rapidly as we are capable of generating more such data for the purpose of training.
> This doesn’t necessarily mean that we’ll see the same sorts of rapid improvements with robot capabilities that we’ve seen with AI more generally, but it does make me think it might be possible.
The key challenge here, I think, is continual learning. LLMs used by agents get by without true learning by starting with trillion-parameter models that have been trained to generalize well across all text-based tasks, then adding million-length token windows with clever compaction/summary techniques.
I think that approach will eventually run out of steam in the physical world. Just too many degrees of freedom, too much noise, too much dependence on the exact situation.
At some point, robots will have to learn on the job. Of course that is a major architecture/algorithm overhaul. Some combination of on-device and cloud training, plus ways to learn new skills without catastrophic forgetting. Plenty of hard problems here, but also plenty of incentive to solve them.
Thanks! That is the most complete description of how these systems work I've seen. May I ask what additional functionality is hoped for by the use of all these enormous data centers that are being constructed. Is it mostly aimed at robotics or what? LLM's by themselves already seem maxed out or very nearly so.
Same architecture as chatbots transformers, SigLIP, Gemma just pointed at actions. The gap isn't the model, it's data text is abundant, robot experience isn't.
The explanation of action chunking helped me picture why the robot needs a sequence of movements, not just one good next move. Running the whole model again for every small action would mean a lot of waiting. That detail made the architecture much more tangible for me.
The VLA name comes from Google DeepMind's RT-2 paper in July 2023. It wrote each robot action out as text tokens, 256 bins per dimension, so one model could learn from web data and robot data together.
There are videos showing that OpenAI's Astra is quite capable of controlling robotic arms. As the LLMs scale they may just keep getting better and better about understanding the 3D world and no specialized model will be needed.
The part I will keep is that the action transformer does not compute a move, it starts from fifty random ones and gets talked out of them. Ten passes, each one pulling the guesses toward something less wrong using whatever the cameras and the instruction and the joint angles say, and what comes out the far end is a sequence of joint targets. That is a very different picture from the one most people carry, where the robot perceives a situation and then decides. It does not decide. It denoises. The reason for the fifty is the detail that makes it real, and it is engineering rather than cognition. One action at a time would mean running the whole stack per movement and the arm would crawl, so the model has to commit to a chunk and live with it. Every operation I have taken apart has a constraint like that sitting underneath behavior that everybody reads as a design philosophy. The number that stopped me was two to the two hundred and fifty sixth. Sixteen pixels by sixteen in black and white alone, and the lookup table is already bigger than the universe, which is the clearest explanation I have seen of why image embeddings have to be a learned function while text embeddings can stay a table. One of those two problems is finite and the other one never was.
The joint training detail is the key one: a 2B-parameter VLM plus a 300M-parameter action transformer, trained together on a billion examples. That undercuts the usual claim that robot data is too scarce to scale.
It feels like the next breakthrough in robotics will depend heavily on data from the physical world: robots need enough real-world experience to learn how to navigate space, handle objects and respond to unpredictable environments.
Hardware then has to catch up. And this is where China may have a real advantage: its manufacturing base gives it a strong path toward making robots cheaper, more scalable and more useful in everyday applications.
The open question is whether Chinese companies will have access to enough high-quality real-world data to train these systems at scale.
Worth stating what the reuse means commercially. A robotics company's capability curve is now set by a 2B parameter backbone somebody else trained on 3T tokens. That is not a technology choice. It is a supplier relationship. The saving is real and so is the dependency. Firms in this position tend to find out what the vendor can charge only after switching has become expensive.
LLMs were able to improve so quickly because there was a huge preexisting corpus of language data that it could be trained with available for (practically) free on the Internet. There is no corpus of physical data in the required format of comparable size (VLAs are not sufficient for general purpose robotics, so images are not enough!), and building one will have a non-trivial cost, and require a great deal of expensive policy coordination. So I think there is reason to believe we will not see the same rapid improvement due to scaling that we saw with LLMs.
Yes, the cognitive burden seems to be extremely high, in other words it requires huge amounts of computation.
Figure Index is an effort to build this training data, and so far they are seeing scaling curves like in LLMs.
Opus 5.5 suggests it’s $.3-.9 million a day paid to the people generating the data, going up 3-6x over the next year: https://claude.ai/share/ed58c43c-813c-46b9-aef0-9252305c9515
The consumer and industrial potential for robotic AI is such that it's not hard to imagine billions being spent to develop massive data on physical/robotic movement. For robots that mimic human movement — the ones that easily capture our interest — the corpus could be something as simple and available as the massive amounts of video data being generated by humans daily.
The important claim that roboticists like Rodney Brooks have made is that video data is not enough - you also need tactile feedback. You can’t learn how to strike a match, or how to safely grip a slippery wet dish, just by looking at where your fingers are - you also have to *feel* what your fingers are feeling.
Having humans teleoperate robots while wearing force feedback gloves may be useful for generating training data, but even figuring out what pattern of sensors the gloves need hasn’t been done.
Yes, if we want to make robots that change diapers or give therapeutic massage, we may need that type of data. But industry, pre-AI, has been producing tactile dexterity in practice for decades. The argument was that the segment of robotic AI that theoretically requires a mass of tactile data was such that it would hold the entire project back. I don't think that's so. And we'll no doubt see huge advances in robotic AI on a commercial level in the coming decade even without the feedback that Brooks and others talk about. And I'll go farther and say that even the data gap will close rapidly as we are capable of generating more such data for the purpose of training.
Nice gentle introduction to the transformer!
> This doesn’t necessarily mean that we’ll see the same sorts of rapid improvements with robot capabilities that we’ve seen with AI more generally, but it does make me think it might be possible.
The key challenge here, I think, is continual learning. LLMs used by agents get by without true learning by starting with trillion-parameter models that have been trained to generalize well across all text-based tasks, then adding million-length token windows with clever compaction/summary techniques.
I think that approach will eventually run out of steam in the physical world. Just too many degrees of freedom, too much noise, too much dependence on the exact situation.
At some point, robots will have to learn on the job. Of course that is a major architecture/algorithm overhaul. Some combination of on-device and cloud training, plus ways to learn new skills without catastrophic forgetting. Plenty of hard problems here, but also plenty of incentive to solve them.
Thanks! That is the most complete description of how these systems work I've seen. May I ask what additional functionality is hoped for by the use of all these enormous data centers that are being constructed. Is it mostly aimed at robotics or what? LLM's by themselves already seem maxed out or very nearly so.
Same architecture as chatbots transformers, SigLIP, Gemma just pointed at actions. The gap isn't the model, it's data text is abundant, robot experience isn't.
The line-segment intuition for why MLPs work clicked for me — I hadn't thought about it as each hidden neuron drawing its own little slope before.
The explanation of action chunking helped me picture why the robot needs a sequence of movements, not just one good next move. Running the whole model again for every small action would mean a lot of waiting. That detail made the architecture much more tangible for me.
The VLA name comes from Google DeepMind's RT-2 paper in July 2023. It wrote each robot action out as text tokens, 256 bins per dimension, so one model could learn from web data and robot data together.
There are videos showing that OpenAI's Astra is quite capable of controlling robotic arms. As the LLMs scale they may just keep getting better and better about understanding the 3D world and no specialized model will be needed.
The part I will keep is that the action transformer does not compute a move, it starts from fifty random ones and gets talked out of them. Ten passes, each one pulling the guesses toward something less wrong using whatever the cameras and the instruction and the joint angles say, and what comes out the far end is a sequence of joint targets. That is a very different picture from the one most people carry, where the robot perceives a situation and then decides. It does not decide. It denoises. The reason for the fifty is the detail that makes it real, and it is engineering rather than cognition. One action at a time would mean running the whole stack per movement and the arm would crawl, so the model has to commit to a chunk and live with it. Every operation I have taken apart has a constraint like that sitting underneath behavior that everybody reads as a design philosophy. The number that stopped me was two to the two hundred and fifty sixth. Sixteen pixels by sixteen in black and white alone, and the lookup table is already bigger than the universe, which is the clearest explanation I have seen of why image embeddings have to be a learned function while text embeddings can stay a table. One of those two problems is finite and the other one never was.
Own robot AI while factories still need the stack.
The joint training detail is the key one: a 2B-parameter VLM plus a 300M-parameter action transformer, trained together on a billion examples. That undercuts the usual claim that robot data is too scarce to scale.
Thanks for the interesting article.
It feels like the next breakthrough in robotics will depend heavily on data from the physical world: robots need enough real-world experience to learn how to navigate space, handle objects and respond to unpredictable environments.
Hardware then has to catch up. And this is where China may have a real advantage: its manufacturing base gives it a strong path toward making robots cheaper, more scalable and more useful in everyday applications.
The open question is whether Chinese companies will have access to enough high-quality real-world data to train these systems at scale.
Without a doubt the single most dangerous threat associated with AI.
A very useful review and primer, nonetheless.
Worth stating what the reuse means commercially. A robotics company's capability curve is now set by a 2B parameter backbone somebody else trained on 3T tokens. That is not a technology choice. It is a supplier relationship. The saving is real and so is the dependency. Firms in this position tend to find out what the vendor can charge only after switching has become expensive.