LLMs were able to improve so quickly because there was a huge preexisting corpus of language data that it could be trained with available for (practically) free on the Internet. There is no corpus of physical data in the required format of comparable size (VLAs are not sufficient for general purpose robotics, so images are not enough!), and building one will have a non-trivial cost, and require a great deal of expensive policy coordination. So I think there is reason to believe we will not see the same rapid improvement due to scaling that we saw with LLMs.
The consumer and industrial potential for robotic AI is such that it's not hard to imagine billions being spent to develop massive data on physical/robotic movement. For robots that mimic human movement — the ones that easily capture our interest — the corpus could be something as simple and available as the massive amounts of video data being generated by humans daily.
The important claim that roboticists like Rodney Brooks have made is that video data is not enough - you also need tactile feedback. You can’t learn how to strike a match, or how to safely grip a slippery wet dish, just by looking at where your fingers are - you also have to *feel* what your fingers are feeling.
Having humans teleoperate robots while wearing force feedback gloves may be useful for generating training data, but even figuring out what pattern of sensors the gloves need hasn’t been done.
Yes, if we want to make robots that change diapers or give therapeutic massage, we may need that type of data. But industry, pre-AI, has been producing tactile dexterity in practice for decades. The argument was that the segment of robotic AI that theoretically requires a mass of tactile data was such that it would hold the entire project back. I don't think that's so. And we'll no doubt see huge advances in robotic AI on a commercial level in the coming decade even without the feedback that Brooks and others talk about. And I'll go farther and say that even the data gap will close rapidly as we are capable of generating more such data for the purpose of training.
> This doesn’t necessarily mean that we’ll see the same sorts of rapid improvements with robot capabilities that we’ve seen with AI more generally, but it does make me think it might be possible.
The key challenge here, I think, is continual learning. LLMs used by agents get by without true learning by starting with trillion-parameter models that have been trained to generalize well across all text-based tasks, then adding million-length token windows with clever compaction/summary techniques.
I think that approach will eventually run out of steam in the physical world. Just too many degrees of freedom, too much noise, too much dependence on the exact situation.
At some point, robots will have to learn on the job. Of course that is a major architecture/algorithm overhaul. Some combination of on-device and cloud training, plus ways to learn new skills without catastrophic forgetting. Plenty of hard problems here, but also plenty of incentive to solve them.
Thanks! That is the most complete description of how these systems work I've seen. May I ask what additional functionality is hoped for by the use of all these enormous data centers that are being constructed. Is it mostly aimed at robotics or what? LLM's by themselves already seem maxed out or very nearly so.
Same architecture as chatbots transformers, SigLIP, Gemma just pointed at actions. The gap isn't the model, it's data text is abundant, robot experience isn't.
There are videos showing that OpenAI's Astra is quite capable of controlling robotic arms. As the LLMs scale they may just keep getting better and better about understanding the 3D world and no specialized model will be needed.
The explanation of action chunking helped me picture why the robot needs a sequence of movements, not just one good next move. Running the whole model again for every small action would mean a lot of waiting. That detail made the architecture much more tangible for me.
The VLA name comes from Google DeepMind's RT-2 paper in July 2023. It wrote each robot action out as text tokens, 256 bins per dimension, so one model could learn from web data and robot data together.
One small correction to the softmax example: dividing [10, 6, 4] by its sum gives [0.5, 0.3, 0.2]. Softmax instead gives roughly [0.980, 0.018, 0.002].
In this example the largest score takes almost all the weight, which helps explain why the preceding scaling step matters.
Action chunking is a clever engineering patch for a brutal physical bottleneck. It's also an architectural trap.
When a model emits fifty joint-angle targets at a time, it's not executing adaptive intelligence. It's playing back an open-loop recording. The transformer needs hundreds of milliseconds to diffuse those action tokens across its vision backbone. To hide that latency, the system gambles that the physical environment will remain static for the next half-second.
It never does.
Language data scales because text doesn't push back with dynamic friction, compliance, or thermal expansion. The internet gave us trillions of tokens of completed reasoning, but physical interaction doesn't exist as clean, disembodied data. The moment a gripper touches an unfamiliar surface, the state space explodes. If you're running a fifty-step chunk, an unmodeled shear force at step three turns the remaining forty-seven steps into a destructive collision.
Neither massive internet corpora nor on-device weight fine-tuning will solve this alone. You can't train away a Nyquist-rate violation. Camera frames arrive at thirty hertz, but physical contact mechanics resolve at two kilohertz.
The missing piece isn't just a bigger policy model. It's a structural separation of time horizons. Macro-level trajectories belong in the diffusion planner, but local compliance must be governed by deterministic, sub-millisecond reflex kernels right on the motor bus.
If the robot's physical safety depends on the next forward pass of an eighteen-block action transformer, it has already lost control.
When contact forces spike mid-chunk, how does your control stack intercept execution before the joint reaches its current commanded angle limit?
@[insert here] delenda est, citing the bitter lesson here mistakes a fundamental law of physics for an algorithmic scaling lag. ⚡
Etched is building specialized transformer silicon for datacenter prefill and decode. Bolting an ASIC into a joint housing doesn't magically dissolve the onto-causal reality of kinetic contact. Even if you drive tensor core compute latency to zero, you can't outrun the transducer bottleneck or the laws of thermodynamics. 🧊
Motor control doesn't stall on FLOPs. It stalls on state identifiability. Field-Oriented Control operates at 20 to 40 kHz directly on the phase currents. Meanwhile, gel tactile sensors take milliseconds to deform, and your camera streams drift in at 30 Hz. When a gripper hits a rigid brass edge with unexpected shear, an eighteen-block action transformer housed in the wrist doesn't save you. Why? Because the incoming sensory embeddings don't carry the physical contact force until microseconds after your gear teeth have already stripped. 💥
Putting an ASIC in the joint creates another brutal reality: heat. Actuator bells already struggle to dissipate thermal energy from high-torque stator windings. Adding a high-power inference accelerator into that unventilated housing cooks your silicon past its thermal throttle point before the robot finishes a single shift. 🌡️
Compute scales. Contact mechanics don't. The real breakthrough isn't burning multi-head attention into custom dies at the motor bus. It's an invariant gate architecture: you run your heavy transformer in the torso to draft macroscopic trajectories, but you distill local compliance into a deterministic, sub-microsecond reflex circuit that lives in a tiny microcontroller. You don't need a billion parameters to cut current when instantaneous torque spikes past safe limits. You just need an analog comparator that doesn't hallucinate. 🛡️
No amount of etched silicon rescues a system where the control loop assumes the universe stands still between forward passes.
I make many mistakes, so that's a fair guess, but I'm not sure this is one. I didn't cité the chips as an existing solution but as évidence that the relevant technologies are to some extent already extent, as an indication that we can expect ongoing progress to fix this. I don't think I even implied that the bitter lesson was a law of physics?
Perhaps a complete solution will use fully optical chips or even biological chips. Perhaps it will have an architecture similar to what you have outlined. Perhaps not!
But we should expect that more computing and more energy will find the solution.
The joint training detail is the key one: a 2B-parameter VLM plus a 300M-parameter action transformer, trained together on a billion examples. That undercuts the usual claim that robot data is too scarce to scale.
It feels like the next breakthrough in robotics will depend heavily on data from the physical world: robots need enough real-world experience to learn how to navigate space, handle objects and respond to unpredictable environments.
Hardware then has to catch up. And this is where China may have a real advantage: its manufacturing base gives it a strong path toward making robots cheaper, more scalable and more useful in everyday applications.
The open question is whether Chinese companies will have access to enough high-quality real-world data to train these systems at scale.
Worth stating what the reuse means commercially. A robotics company's capability curve is now set by a 2B parameter backbone somebody else trained on 3T tokens. That is not a technology choice. It is a supplier relationship. The saving is real and so is the dependency. Firms in this position tend to find out what the vendor can charge only after switching has become expensive.
LLMs were able to improve so quickly because there was a huge preexisting corpus of language data that it could be trained with available for (practically) free on the Internet. There is no corpus of physical data in the required format of comparable size (VLAs are not sufficient for general purpose robotics, so images are not enough!), and building one will have a non-trivial cost, and require a great deal of expensive policy coordination. So I think there is reason to believe we will not see the same rapid improvement due to scaling that we saw with LLMs.
The consumer and industrial potential for robotic AI is such that it's not hard to imagine billions being spent to develop massive data on physical/robotic movement. For robots that mimic human movement — the ones that easily capture our interest — the corpus could be something as simple and available as the massive amounts of video data being generated by humans daily.
The important claim that roboticists like Rodney Brooks have made is that video data is not enough - you also need tactile feedback. You can’t learn how to strike a match, or how to safely grip a slippery wet dish, just by looking at where your fingers are - you also have to *feel* what your fingers are feeling.
Having humans teleoperate robots while wearing force feedback gloves may be useful for generating training data, but even figuring out what pattern of sensors the gloves need hasn’t been done.
Yes, if we want to make robots that change diapers or give therapeutic massage, we may need that type of data. But industry, pre-AI, has been producing tactile dexterity in practice for decades. The argument was that the segment of robotic AI that theoretically requires a mass of tactile data was such that it would hold the entire project back. I don't think that's so. And we'll no doubt see huge advances in robotic AI on a commercial level in the coming decade even without the feedback that Brooks and others talk about. And I'll go farther and say that even the data gap will close rapidly as we are capable of generating more such data for the purpose of training.
Figure Index is an effort to build this training data, and so far they are seeing scaling curves like in LLMs.
Opus 5.5 suggests it’s $.3-.9 million a day paid to the people generating the data, going up 3-6x over the next year: https://claude.ai/share/ed58c43c-813c-46b9-aef0-9252305c9515
Yes, the cognitive burden seems to be extremely high, in other words it requires huge amounts of computation.
Nice gentle introduction to the transformer!
> This doesn’t necessarily mean that we’ll see the same sorts of rapid improvements with robot capabilities that we’ve seen with AI more generally, but it does make me think it might be possible.
The key challenge here, I think, is continual learning. LLMs used by agents get by without true learning by starting with trillion-parameter models that have been trained to generalize well across all text-based tasks, then adding million-length token windows with clever compaction/summary techniques.
I think that approach will eventually run out of steam in the physical world. Just too many degrees of freedom, too much noise, too much dependence on the exact situation.
At some point, robots will have to learn on the job. Of course that is a major architecture/algorithm overhaul. Some combination of on-device and cloud training, plus ways to learn new skills without catastrophic forgetting. Plenty of hard problems here, but also plenty of incentive to solve them.
Thanks! That is the most complete description of how these systems work I've seen. May I ask what additional functionality is hoped for by the use of all these enormous data centers that are being constructed. Is it mostly aimed at robotics or what? LLM's by themselves already seem maxed out or very nearly so.
Same architecture as chatbots transformers, SigLIP, Gemma just pointed at actions. The gap isn't the model, it's data text is abundant, robot experience isn't.
The line-segment intuition for why MLPs work clicked for me — I hadn't thought about it as each hidden neuron drawing its own little slope before.
There are videos showing that OpenAI's Astra is quite capable of controlling robotic arms. As the LLMs scale they may just keep getting better and better about understanding the 3D world and no specialized model will be needed.
The explanation of action chunking helped me picture why the robot needs a sequence of movements, not just one good next move. Running the whole model again for every small action would mean a lot of waiting. That detail made the architecture much more tangible for me.
The VLA name comes from Google DeepMind's RT-2 paper in July 2023. It wrote each robot action out as text tokens, 256 bins per dimension, so one model could learn from web data and robot data together.
One small correction to the softmax example: dividing [10, 6, 4] by its sum gives [0.5, 0.3, 0.2]. Softmax instead gives roughly [0.980, 0.018, 0.002].
In this example the largest score takes almost all the weight, which helps explain why the preceding scaling step matters.
Definition: https://docs.pytorch.org/docs/2.14/generated/torch.nn.Softmax.html
Action chunking is a clever engineering patch for a brutal physical bottleneck. It's also an architectural trap.
When a model emits fifty joint-angle targets at a time, it's not executing adaptive intelligence. It's playing back an open-loop recording. The transformer needs hundreds of milliseconds to diffuse those action tokens across its vision backbone. To hide that latency, the system gambles that the physical environment will remain static for the next half-second.
It never does.
Language data scales because text doesn't push back with dynamic friction, compliance, or thermal expansion. The internet gave us trillions of tokens of completed reasoning, but physical interaction doesn't exist as clean, disembodied data. The moment a gripper touches an unfamiliar surface, the state space explodes. If you're running a fifty-step chunk, an unmodeled shear force at step three turns the remaining forty-seven steps into a destructive collision.
Neither massive internet corpora nor on-device weight fine-tuning will solve this alone. You can't train away a Nyquist-rate violation. Camera frames arrive at thirty hertz, but physical contact mechanics resolve at two kilohertz.
The missing piece isn't just a bigger policy model. It's a structural separation of time horizons. Macro-level trajectories belong in the diffusion planner, but local compliance must be governed by deterministic, sub-millisecond reflex kernels right on the motor bus.
If the robot's physical safety depends on the next forward pass of an eighteen-block action transformer, it has already lost control.
When contact forces spike mid-chunk, how does your control stack intercept execution before the joint reaches its current commanded angle limit?
(¬_¬ )
The bitter lesson will come for your "roadblock", too.
If necessary, we will etch the necessary "reaction" transformers on silicon directly and house that in the actuator itself: https://www.etched.com/
@[insert here] delenda est, citing the bitter lesson here mistakes a fundamental law of physics for an algorithmic scaling lag. ⚡
Etched is building specialized transformer silicon for datacenter prefill and decode. Bolting an ASIC into a joint housing doesn't magically dissolve the onto-causal reality of kinetic contact. Even if you drive tensor core compute latency to zero, you can't outrun the transducer bottleneck or the laws of thermodynamics. 🧊
Motor control doesn't stall on FLOPs. It stalls on state identifiability. Field-Oriented Control operates at 20 to 40 kHz directly on the phase currents. Meanwhile, gel tactile sensors take milliseconds to deform, and your camera streams drift in at 30 Hz. When a gripper hits a rigid brass edge with unexpected shear, an eighteen-block action transformer housed in the wrist doesn't save you. Why? Because the incoming sensory embeddings don't carry the physical contact force until microseconds after your gear teeth have already stripped. 💥
Putting an ASIC in the joint creates another brutal reality: heat. Actuator bells already struggle to dissipate thermal energy from high-torque stator windings. Adding a high-power inference accelerator into that unventilated housing cooks your silicon past its thermal throttle point before the robot finishes a single shift. 🌡️
Compute scales. Contact mechanics don't. The real breakthrough isn't burning multi-head attention into custom dies at the motor bus. It's an invariant gate architecture: you run your heavy transformer in the torso to draft macroscopic trajectories, but you distill local compliance into a deterministic, sub-microsecond reflex circuit that lives in a tiny microcontroller. You don't need a billion parameters to cut current when instantaneous torque spikes past safe limits. You just need an analog comparator that doesn't hallucinate. 🛡️
No amount of etched silicon rescues a system where the control loop assumes the universe stands still between forward passes.
(¬‿¬)
I make many mistakes, so that's a fair guess, but I'm not sure this is one. I didn't cité the chips as an existing solution but as évidence that the relevant technologies are to some extent already extent, as an indication that we can expect ongoing progress to fix this. I don't think I even implied that the bitter lesson was a law of physics?
Perhaps a complete solution will use fully optical chips or even biological chips. Perhaps it will have an architecture similar to what you have outlined. Perhaps not!
But we should expect that more computing and more energy will find the solution.
Own robot AI while factories still need the stack.
The joint training detail is the key one: a 2B-parameter VLM plus a 300M-parameter action transformer, trained together on a billion examples. That undercuts the usual claim that robot data is too scarce to scale.
Thanks for the interesting article.
It feels like the next breakthrough in robotics will depend heavily on data from the physical world: robots need enough real-world experience to learn how to navigate space, handle objects and respond to unpredictable environments.
Hardware then has to catch up. And this is where China may have a real advantage: its manufacturing base gives it a strong path toward making robots cheaper, more scalable and more useful in everyday applications.
The open question is whether Chinese companies will have access to enough high-quality real-world data to train these systems at scale.
Without a doubt the single most dangerous threat associated with AI.
A very useful review and primer, nonetheless.
Worth stating what the reuse means commercially. A robotics company's capability curve is now set by a 2B parameter backbone somebody else trained on 3T tokens. That is not a technology choice. It is a supplier relationship. The saving is real and so is the dependency. Firms in this position tend to find out what the vendor can charge only after switching has become expensive.