The enormous advance in AI capabilities over the last several years has in large part been driven by the availability of vast amounts of training data. Gwern’s influential 2020 essay “The Scaling Hypothesis,” written after the release of GPT-3, argues that recent advances in AI have been the result of scale: bigger models, bigger datasets, and more compute spent training them. Gwern writes:
The scaling hypothesis regards the blessings of scale as the secret of AGI: intelligence is ‘just’ simple neural units & learning algorithms applied to diverse experiences at a (currently) unreachable scale. As increasing computational resources permit running such algorithms at the necessary scale, the neural networks will get ever more intelligent.
Likewise, pseudonymous AI researcher Nostalgebraist wrote in 2024 that “the entire current landscape dates back to the realization that ML models improve steadily and predictably as you train them on more data. And that everything since then consists of training bigger models on larger datasets, enabled by years of rapid improvement in hardware.” Epoch AI found that since the early 2010s, the size of training datasets for AI language models has doubled about every six months. The biggest AI models today are trained on tens of trillions of tokens.
As more and more effort gets devoted to applying modern AI to robotics, an important question is “where will the training data come from?” Language models could be trained on huge existing corpuses of text. GPT-3, for instance, was trained on about 300 billion tokens’ worth of data scraped from the internet. In 2024 Epoch AI estimated the total existing stock of quality, accessible text data is around 300 trillion tokens. But there’s no equivalent stock of good, public robot training data. Figuring out where the data for robotics will come from and how much of it we’ll need has become a key problem for the field.
What is robot training data?
Training an AI model, whether it’s an LLM or a model used for controlling robots, fundamentally requires lots and lots of examples of something being done correctly (or enough knowledge of what a correct action looks like that you can score an attempt). You feed a model some set of inputs, you compare the model’s output to the actual, correct output, and you use the difference between the two to modify the parameters of the model to reduce this difference via some flavor of gradient descent. (We went into this process in some depth earlier this month.)
With text, a training example can be just an existing chunk of text; you take the chunk, feed the model part of it, and have the model predict what comes next in the sequence. One chunk of text can thus generate many training examples by feeding the model different portions of the text.
To train a robot, an example is typically some sort of recording of a task being accomplished or attempted. There are a variety of different methods used to create these recordings and turn them into training data.
Historically, one of the most common methods for creating these recordings is called “teleoperation.” With teleoperation, a person manipulates some controls that are directly connected to a robot, and uses them to move a robot through some particular task (folding a shirt, picking up and moving objects, and so on). The image below, for instance, shows a person teleoperating a pair of robot arms, using them to pour water from a pitcher into a glass.
As the person performs the task, data of the robot’s state is recorded, along with images from cameras that are recording the task (often mounted to the robot itself), which are combined to create a training example. An example might consist of feeding a model some data on the robot’s state and what the cameras can see at some particular point in the sequence. The model will try to predict what the next robot actions are (in the form of something like joint angles), and then compare that to what the actual actions were. This sort of training strategy, using examples generated by an expert (a human) and then training a model to mimic them, is known as behavioral cloning, and it’s part of the broader category of “imitation learning.”
The major benefit of this method is that the data it generates very closely matches the robot’s actual movement capabilities, because it’s produced from the actual robot you want to train, and it remains widely used today. You sometimes see videos or pictures from huge data collection factories, filled with people doing nothing but teleoperating robots to complete various tasks, and you sometimes see teleoperation accidents when the person forgets that they’re controlling the robot and suddenly moves in a way that the robot can’t handle.
But teleoperation has drawbacks. The biggest one is probably expense: to get the data, you need to have an actual robot, along with a person to sit there and manipulate it, which makes it hard to scale this method to produce very large amounts of data. Another drawback is that it’s somewhat clunky for a person to control a robot this way; they often can’t get any sort of tactile feedback from the object being manipulated to guide their motions, so this won’t necessarily produce really good data on dexterous manipulation. The clunkiness can also make it hard to collect good teleoperation data, and as I understand it, only a fraction of total teleoperation data collected ends up being useful for training.
It’s also possible to generate training data using an existing robot by having it move around on its own, recording its actions, and using some sort of system to categorize these behavioral sequences as successful or not. This sort of strategy, where the robot moves autonomously and you score its actions in some way, is known as “reinforcement learning.” Google was doing something like this around 2016 with an “arm farm”: a series of robot arms moving autonomously, with success scored by whether an arm picked up or grasped an object or not.
This sort of arm farm doesn’t seem to be widely used today, but using real-world deployed robots will probably eventually be the largest source of robot training data. You often hear robotics folks talk about the need for a “data flywheel,” where you have robots deployed in the real world, you use the data from them to train even better models, resulting in more robot deployments, resulting in better models, and so on.
Robotic AI company Physical Intelligence, for instance, is experimenting with this sort of “deploy a robot to do real work, and use it to gather data to improve the model” proto-flywheel at a chocolate factory near its offices. But while this might work eventually, right now there’s a very small number of AI-controlled robots doing anything resembling real work. Real-world data also tends to be extremely repetitive — robots doing the same task over and over again in very similar conditions — which makes it much less useful for training. So for now most data needs to come from other sources.
Another very common strategy for generating robotic training data is simulation: instead of running a physical robot in the real world, run a digital one in a simulated world instead. This is also commonly done with reinforcement learning, in which you have the robot try to navigate some simulated space, after which you score it on how well it does. Another flavor of this is the teacher/student paradigm where you have a “teacher” model that has full access to all the data within the simulated environment, and a “student” model that can only see what the real robot would see (images from its cameras and its own internal state), and you try to get the student to mimic the teacher.
The benefits of simulation are that it’s much cheaper and faster to simulate a robot than it is to deploy one in the real world, and that you can get very large amounts of data by simulating the behavior of thousands of robots. Figure AI taught its humanoid robot to walk via reinforcement learning on thousands of simulated robots. OpenAI taught a robotic hand to solve a Rubik’s Cube via the equivalent of 13,000 years of simulated experience. By 2019, Waymo had driven 10 million miles in the real world, but 10 billion miles in simulation.
But simulation also has drawbacks. The biggest one is that the quality of the data it produces is a function of the accuracy of the simulation. For some things, such as locomotion (having a robot move through some physical space), simulation is fairly accurate, because the behavior is mostly well-characterized enough that we can represent it digitally. So you can train a quadruped robot to navigate some terrain, or a bipedal robot to dance, or a drone to fly along some path in simulation, and the simulated environment will be a reasonably good approximation of the real world. But for things such as object manipulation (particularly for soft or deformable objects), it’s much harder to build a simulation that accurately captures real-world behavior, and using it as a source of training data is much more difficult. This is generally referred to as the “sim-to-real gap.”
Because of simulation inaccuracies, one popular simulation technique is called “domain randomization.” Since you don’t know what various parameters like friction will be in the real world, you decide to vary the parameters randomly when running the simulation, in the hopes of creating a training envelope that encompasses the parameters that will be encountered in the real world. When OpenAI trained its robotic hand to solve a Rubik’s Cube, for instance, it continuously expanded the range of parameters being tested, forcing the model to learn to solve the cube in more and more difficult environments, in the hopes of giving it capabilities that would generalize to the real world. The robotics team there used this strategy specifically because accurate physical modeling was too difficult:
The biggest challenge we faced was to create environments in simulation diverse enough to capture the physics of the real world. Factors like friction, elasticity and dynamics are incredibly difficult to measure and model for objects as complex as Rubik’s Cubes or robotic hands and we found that domain randomization alone is not enough.
To overcome this, we developed a new method called Automatic Domain Randomization (ADR), which endlessly generates progressively more difficult environments in simulation. This frees us from having an accurate model of the real world, and enables the transfer of neural networks learned in simulation to be applied to the real world.
But despite this technique, and really quite a lot of simulation (13,000 simulated years), performance was only OK — for maximally difficult starting arrangements, success was only 20% — which is maybe why this technique doesn’t seem to be widely used for manipulation training currently. Environmental parameter randomization does seem to be fairly standard for locomotion training though, used for things like randomly varying the coefficient of friction of the surface being traversed.
Another way to use simulation is to create synthetic training data: taking real training data as an input and using it to create lots of similar training examples. Nvidia, for instance, has a page showing how you can use its Isaac GR00T robot simulation platform to generate synthetic robot training data using teleoperation data as an input. Synthetic data generation also seems to be widely used for training autonomous vehicles.
We can also create robot training data by recording humans doing tasks and creating some sort of mapping from the recorded task data to a robot. This might be as simple as fitting a person with a head-mounted camera (creating what’s known as “egocentric video”), and using it to record the person doing various tasks with their hands. The video is then processed so the person’s motions are interpreted as robot motions, and used to create training examples. (This is one reason for the popularity of the humanoid form factor in robotics: humans doing tasks will be a reasonably good source of training data.) Figure AI, for instance, recently announced Index, its program for gathering training data by sending people head-mounted cameras to record themselves doing various tasks (alternatively, Figure will also send a crew equipped with these cameras to your home or business to do chores for you). Companies like Shift are also doing this, sending people with hat-mounted cameras to folks’ houses to record themselves doing chores.
You can also add more sensors to collect more elaborate human task data. The now-defunct startup Eidon AI stuck sensors on folks’ arms to collect data on joint positions for training robot arms. Neocambrian AI recently announced “Mitts,” which it describes as “state of the art tactile gloves that capture high-precision touch data for robots.”
Another flavor of this strategy is to use what’s called a Universal Manipulation Interface, or UMI. With UMI, a person holds a pair of specially designed manipulators, that typically have a simple pincer grip at the end (though some have multiple fingers). The person does various tasks with the manipulators, which have cameras mounted to them that record exactly what the manipulators are doing. This video is then used to create training examples for robots that have similar manipulators. Robot model startups Generalist AI and Sunday Robotics, for instance, seem to be primarily using this strategy for gathering training data.
The benefit of using human-collected task data is that it’s much cheaper to collect than teleoperation data: all you need is a camera, or some camera-equipped manipulators, rather than an entire robot. And it’s probably much easier to get a wide variety of different task recordings by just having a person walk around and do different things, rather than constantly repositioning a robot to do them. And because the data is gathered in the real world, by people manipulating real objects, it’s more accurate than simulation. The drawback is that because this data comes from a person rather than a robot, it won’t map onto a robot’s capabilities perfectly well. Even UMI, which minimizes the human-robot differences by having a person do tasks using a sort of robotic manipulator, won’t necessarily map perfectly, though the discrepancy is small and many of the most impressive robotics demos we’ve seen (such as Generalist AI’s) have been trained using UMI data.
It’s possible to use multiple training strategies in a single robot model. Agility Robotics’ Digit, for instance, walks using a small, 1-million-parameter neural network trained in simulation using reinforcement learning, while object manipulation seems to have been trained using behavioral cloning. Dyna Robotics similarly seems to have used a combination of reinforcement learning in simulation and human video for training data.
It’s also often useful to train a robot model using data other than task completion data. Vision-Language-Action models, as we noted earlier, are built from Vision-Language models trained on text and images. And because of the ubiquity of general video data on things like YouTube or TikTok, there’s a great deal of interest in finding ways to use generic web videos as a source of training data. Nvidia’s Cosmos, for instance, is a foundation model trained on video data that can be used as a backbone for a robot policy model.
Robot dataset sizes
How much of this action-relevant training data have we gathered so far, and how much do we need?
As with AI models more generally, robotic training datasets seem to be getting bigger. An early dataset that was considered large at the time was the Open X-Embodiment (OXE) dataset, created by Google DeepMind in 2023 by stitching together dozens of existing datasets. OXE consisted of over 1 million robot action trajectories spanning more than 160,000 different tasks, totaling around 4,000 hours’ worth of task data. Physical Intelligence’s Pi 0 model, released in October of 2024, was trained on 10,000 hours of robot task data. Generalist AI’s GEN 1 model, announced earlier this year, was trained on around 500,000 hours of UMI data. Figure AI’s Index announcement stated that it was gathering 30 minutes of video data every second, or 43,000 hours a day. Dyna Robotics and Nature Will both claim that their latest models are trained on over 1 million hours of human egocentric video data. I saw one estimate that the robot data collection industry is collectively producing around 500,000 hours of training data a month.
It’s not amazingly easy to compare these datasets to the text datasets used to train LLMs. For one, different robotic training sets have different levels of quality — teleoperation or UMI data is probably higher quality than pure human egocentric video data for training purposes, and a highly variable dataset with lots of different sorts of tasks is much more useful than an extremely repetitive one. And video data for robotics training is informationally different from text data: for one, video data tends to be highly correlated — on short timescales, one frame is pretty similar to another frame. Thus even though video recordings contain a lot of data, they’re probably less informationally dense for training purposes than text data. As robotics researcher Chris Paxton notes, “machine learning works best the closer you are to an independent and identically distributed (iid) dataset; neither web data nor robot data is iid, but robot data is far less iid than web data.”
But we can still try. One very rough way to compare dataset sizes, used by Physical Intelligence researcher Kevin Black in this presentation and robotics professor Ken Goldberg in this presentation, is to estimate how long a dataset of text tokens would take for a person to read, and compare that to the hours of robot training data we have. Under this metric, GPT-2 was trained on the equivalent of around 475,000 hours of data, GPT-3 was trained on 26 million hours, and GPT-4 was trained on 685 million hours. Using this (somewhat dubious) metric, even several-year-old AI models were trained on something like 1,000 times the data that current robot models are trained on. Goldberg calls this the “100,000-year data gap.”
On his Substack “It Can Think!” Paxton makes an estimate from the other direction, converting hours of robot data into tokens, and comes up with a fairly similar estimate. He guesses that for video running at 10 frames per second, each frame will contain the equivalent of one useful training token. Every hour of video thus contains about 10 × 60 × 60 = 36,000 training tokens. He then divides this by 10, on the assumption that you can’t expect every second of video data to actually be useful, getting you about 3,600 training tokens per hour of video. This would make Dyna Robotics’s 1 million hours of training equal to around 3.6 billion tokens, on the order of 5,000 times smaller than the datasets used to train recent AI models.
If you assume that we need about 1,000 times as much real-world robot training data as we actually have — on the order of 1 billion hours of data — how much would that cost to generate? Eidon AI reports robot data prices of $60 to $100 per hour. Pantheon similarly reports UMI data costs from $60 to $150 per hour, but that its engineers developed a pipeline capable of generating data for $10 per hour. Shift claims you can earn up to $20 per hour recording egocentric video data and selling it to them. So a naive estimate suggests that getting enough data to get robot models to a similar level of capability as recent AI models might cost on the order of $10 billion to $100 billion.
How much data we actually need, however, seems to be something of an open question. If we don’t care about generalization, and only want to train robot models to perform various specific tasks, the data requirements might be substantially less. Alternatively, it might turn out that frontier general AI models like Fable or Astra are extremely capable robot controllers after being fed a comparatively small amount of robotic training data; some folks are already predicting that special-purpose robot AI models might get replaced with more generic AI models. Likewise, we might find robotic AI model architectures that are much more sample efficient — learning much more from each training example — than current models.
On the other hand, it could be the case that moving around and manipulating objects in the real world is a fundamentally more difficult task than manipulating text on a computer, and that models require a lot more training to become competent at this real-world behavior. Small errors in generated text, for instance — misspellings, or slightly wrong word choice — don’t necessarily compromise the text as a whole, but when you’re manipulating objects a single tiny positioning error can be the difference between success and complete failure. Likewise, Ken Goldberg argues that robotic movement is a much higher-dimensional problem than generating text (though I’m not sure I buy the version of the argument he articulates in that presentation, but I have seen this sentiment echoed by other robotics researchers).
Conclusion
LLMs showed the benefits of scale when training AI models — bigger models, more data, trained using much more computation. We’re now in the process of seeing how well this scaling works for robotics, and robotics researchers are working hard to gather enormous amounts of robot task training data in the hopes of seeing the same sort of trends with robotic capabilities that we have with AI more generally.
One sensor-side asymmetry behind your cost-per-hour numbers: robot vision rides on phone-camera volume, touch doesn't. Sony and TSMC's ¥747bn image-sensor venture in Kumamoto is built for smartphone sensors, with "physical AI" (cars, robotics) listed as an area to explore. Tactile has no volume market like that, and contact sensors wear out. Touch may end up being the expensive hours.