The State of Data Collection for Robotic Manipulation

A beginner-friendly guide to teleoperation, human demonstrations, simulation, data gloves, tactile sensing, and the trade-offs behind scalable robot learning.

Preface

During my years in humanoid robotics, I was not particularly focused on data collection methods and practices. However, over the past two years, several projects pushed me to look more closely at this area and connect it with my broader experience in robotics. More recently, I have encountered many newcomers and beginners who are curious about how manipulation data is actually collected, what the main approaches are, and why the topic has become so important. That made me realize there is a need for a beginner-friendly piece that is still as detailed and extensive as possible.

This article is my attempt to fill that gap. It covers the main ways manipulation data is collected including teleoperation, human demonstrations, portable devices, simulation, multi-robot datasets, tactile sensing, and autonomous robot experience. The goal is not only to describe these methods, but also to explain what each one captures well, what it misses, and where the trade-offs appear in practice.

The main conclusion is that there is no single best data source. Teleoperation provides robot-executable actions but is expensive, human video scales well but creates an embodiment gap, and simulation offers large-scale data at the cost of imperfect realism. In practice, the most useful systems are likely to rely on a combination of broad human and synthetic data, diverse robot datasets, and smaller amounts of high-quality data collected directly from the target robot.

I hope this piece gives newcomers a clearer starting point and helps more experienced readers organize the different ideas around manipulation data collection into one practical picture. The field is moving quickly, and many of the current methods will continue to evolve, but the central question will remain the same: not simply how much data we can collect, but whether we are collecting the information the robot actually needs to understand, act, and recover in the real world.

Disclaimer: This article is based on personal experience and observations, as well as various sources and experiments. It was originally written as three shorter articles that were later combined into one. The final version was proofread and lightly adjusted with AI to improve clarity and readability.

More cameras, more robots, more demonstrations, and still no free lunch

Robotics has a data problem, but it is not simply that we need more data.

We need the right data, collected in the right places, with the right sensors, using actions that can actually be transferred to a robot. That distinction matters because 10,000 hours of human video, 10,000 robot demonstrations, and 10,000 simulated trajectories are not interchangeable. They teach different things.

A human wearing a head-mounted camera can generate enormous visual diversity. They can walk into a new kitchen, pick up an unfamiliar tool, open a drawer, fold a shirt, and move naturally around clutter. But the camera does not directly record the motor command that a robot should execute. It also does not directly measure grip force, friction, joint torque, or the pressure distribution between the hand and the object.

A teleoperated robot demonstration has the opposite profile. The data already lives in the robot’s action space. Joint states, gripper commands, camera images, and force readings can all be synchronized. The demonstration can be used almost directly for imitation learning. But collecting it is slow, expensive, physically constrained, and usually tied to a particular robot.

Simulation can produce millions of perfectly labeled trajectories without damaging hardware or paying operators. It also produces millions of trajectories inside a world that is not quite real.

This is the central issue in manipulation data collection:

Every collection method gives us certain signals cheaply and hides other signals completely.

There is no universal best source. The useful question is not, “Which data collection method wins?” It is, “Which uncertainty am I willing to accept, and which one will break my task?”

That is what this article is about.

First, what exactly counts as robot manipulation data?

A manipulation dataset is usually organized into episodes, sometimes called trajectories or demonstrations. An episode records a robot, human, or simulated agent attempting a task over time.

A simplified episode might look like this:

[
\tau = {(o_t, a_t)}_{t=1}^{T}
]

where:

• (o_t) is the observation at time (t)

• (a_t) is the action taken at time (t)

• (T) is the duration of the episode

The observation may include:

• RGB images

• depth images

• wrist-camera images

• joint positions

• joint velocities

• end-effector pose

• gripper position

• measured joint torques

• force-torque sensor readings

• tactile images or pressure values

• audio (rare case, IMO)

• language instructions

• object poses

• calibration parameters

The action may be represented as:

• desired joint positions

• joint velocities

• joint torques

• absolute end-effector poses

• changes in end-effector pose

• gripper-open or gripper-close commands

• continuous finger-joint targets

• a chunk of several future actions

A language-conditioned dataset might also associate the trajectory with an instruction:

Put the red cup into the sink.

A more richly annotated dataset may include:

• whether the attempt succeeded

• the moment at which contact occurred

• the object being manipulated

• the skill being performed

• the cause of failuree

• recovery actions

• subtask boundaries

• operator identity

• robot embodiment

• environmental conditions

The fact that all these things are called “robot data” creates confusion. A video of someone folding a towel and a time-synchronized robot trajectory of a gripper folding a towel may look similar to a human viewer. From the perspective of a learning algorithm, they are very different supervision signals.

Why manipulation data is unusually difficult

Language models can learn from text that already exists. Vision models can learn from photographs and videos that people upload naturally.

Robot manipulation does not have an equivalent internet-scale corpus.

A robot action is not just an image or a description. It is a physical intervention. Someone or something must move an actuator, make contact with an object, experience the resulting dynamics, and record the process.

This creates several problems.

Robot time is expensive

A physical robot must be purchased, installed, calibrated, maintained, supervised, and reset between attempts. Grippers wear out. Cameras move. Objects break. Cables disconnect. Operators make mistakes. A demonstration that lasts 20 seconds may require another minute to prepare.

The actual cost is rarely the duration of the trajectory. The expensive part is everything around it.

The robot changes the data

A human can reach behind an object, slide a finger under a thin sheet, or reorient the wrist through a narrow space. A parallel-jaw gripper cannot necessarily do those things.

The hardware determines which demonstrations are possible. This is called the embodiment problem.

A demonstration collected with one arm may not transfer cleanly to another arm because the robots differ in:

• reach

• joint limits

• kinematics

• control frequency

• payload

• gripper geometry

• compliance

• camera placement

• base mobility

For dexterous hands, the gap becomes even larger. Human hands have different joint structures, soft tissue, sensing, passive compliance, and contact geometry from robotic hands.

The most important moments are often occluded

During contact-rich manipulation, the hand or gripper often blocks the camera’s view of the relevant area.

Consider inserting a plug, turning a key, wiping a surface, tying a knot, or sliding one object against another. The camera may show the overall motion, but the state that determines success can depend on:

• contact force

• friction

• pressure distribution

• micro-slip

• deformation

• alignment errors smaller than a pixel

• whether an object is fully seated

Vision may infer some of these variables from motion, deformation, sound, or context. It does not directly observe them.

Success data is easier to collect than recovery data

Demonstration datasets are usually biased toward successful behavior. Operators repeat a task until it looks clean, then save the successful attempt.

This produces attractive training data but an incomplete state distribution.

A policy trained only on successful demonstrations sees what to do when everything is going well. During deployment, a small error can move it into a state that never appeared in the demonstrations. The policy then makes another error, moves even farther from the training distribution, and eventually fails.

This is the classic covariate-shift problem in behavior cloning.

Failure, correction, and recovery data are less visually impressive, but they may be more valuable than another hundred nearly identical successful demonstrations.

Dataset size is difficult to compare

One paper reports hours. Another reports episodes. Another reports frames.

None of these numbers are directly comparable.

A dataset containing one million five-second clips is not automatically larger, in a useful sense, than a dataset containing 50,000 two-minute trajectories. One dataset may record at 5 Hz and another at 60 Hz. Some datasets contain repeated versions of one task, while others contain hundreds of tasks with only a few demonstrations each.

The raw number should always be followed by several questions:

• How long is an average trajectory?

• How many unique tasks are represented?

• How many objects, rooms, and operators appear?

• How much of the data is successful?

• Are actions available?

• Are the actions robot-executable?

• How accurately are sensors synchronized?

• Does the test set contain genuinely unseen conditions?

Scale matters, but only after defining what was scaled.

The main ways manipulation data is collected

• Simulation/ Synthetic Data

• Egocentric Human Video

• Federated existing robot dataset

• Portable proxy device (eg. UMI)

• Distributed real-robot collection

• Data Gloves/ haptic gloves

• Autonomous robot experience (real)

• Kinematic Teaching

• Direct Teleoperation

Below, I’ll try to explain them extensively:

Press enter or click to view image in full size

1. Direct teleoperation

Teleoperation means that a human controls the physical robot while the robot records observations and actions.

Common interfaces include:

• keyboard or joystick control

• 3D SpaceMouse devices

• virtual-reality controllers

• motion-capture systems

• exoskeletons

• leader-follower arms

• miniature replicas of the target robot

• bilateral haptic interfaces

The main advantage is alignment.

The human does not merely show what the task looks like. They generate actions through the target robot. The resulting dataset can contain exactly the variables the policy will later need to predict.

For example, a trajectory might contain:

[
a_t =
[\Delta x,\Delta y,\Delta z,\Delta r,\Delta p,\Delta yg]
]

where the first three values represent Cartesian translation, the next three represent orientation change, and (g) represents the gripper command.

If the deployment policy will output that same action representation, training becomes relatively straightforward.

SpaceMouse and VR control

End-effector teleoperation interfaces map human controller movement into a desired robot-hand pose.

They are flexible and comparatively inexpensive. The same controller can often operate different robots.

Their weakness is that the operator does not directly feel the robot’s kinematic configuration. The controller says where the hand should move, but not necessarily how the arm should arrange itself. Singularities, joint limits, self-collisions, and awkward elbow configurations may be difficult for the operator to anticipate.

Latency is another problem. Even 100 to 200 milliseconds can make precise contact feel disconnected. Operators compensate by moving more slowly and cautiously than they would during a natural demonstration.

Leader-follower systems

A leader arm has a similar or identical kinematic structure to the robot being controlled. Moving the leader produces corresponding motion on the follower robot.

GELLO is a well-known example. It uses a low-cost, scaled controller whose kinematics match the target arm. In its user study, GELLO produced more reliable and efficient demonstrations than other inexpensive interfaces such as VR controllers and SpaceMouse devices. The system was demonstrated with Franka, UR5, and xArm robots, including bimanual and contact-rich tasks.

This style of interface reduces the mental translation required from the operator. The person moves something shaped like an arm rather than moving a floating cursor in six-dimensional space.

The trade-off is hardware specificity. A controller designed around one arm does not automatically fit another arm.

ALOHA and Mobile ALOHA

ALOHA-style systems use smaller leader arms to teleoperate two follower arms. They became popular because they made relatively low-cost bimanual data collection practical.

Mobile ALOHA extended the setup with a mobile base and whole-body teleoperation. The system was used to collect demonstrations for tasks such as cooking, cabinet manipulation, elevator operation, and rinsing a pan. The work also showed that co-training on static ALOHA data could improve mobile-manipulation performance, even when only 50 task-specific demonstrations were available.

This highlights an important pattern: a small amount of highly aligned, task-specific data can become much more useful when combined with broader pretraining data.

Pros of direct teleoperation

• Actions are already expressed through a robot.

• Robot state and sensor data can be recorded directly.

• Contact and dynamics correspond to the real hardware.

• Demonstrations respect the robot’s reach and joint limits.

• Data can be used directly for behavior cloning or offline reinforcement learning.

• It is relatively easy to add force, torque, tactile, or audio sensing.

Cons of direct teleoperation

• Collection speed is limited by robot speed.

• Every trajectory consumes physical hardware time.

• Resetting tasks can dominate total collection time.

• Operators require training.

• Bad interfaces produce awkward or low-quality demonstrations.

• Data is often specific to one embodiment.

• Hardware failures interrupt collection.

• Safety constraints reduce exploration.

• Collecting in many homes, factories, or public environments is logistically difficult.

Teleoperation remains the cleanest source of executable robot actions. It is also one of the hardest methods to scale geographically and economically.

Press enter or click to view image in full size

2. Kinesthetic teaching

In kinesthetic teaching, a human physically moves the robot arm through the desired motion.

This can work well with lightweight, backdrivable, or gravity-compensated arms. The operator guides the end effector directly while the robot records joint or Cartesian trajectories.

It is intuitive for simple paths:

• moving between waypoints

• opening a drawer

• placing an object

• guiding a wiping motion

• teaching an insertion direction

It becomes less effective when the task requires:

• coordinated bimanual motion

• dynamic actions

• continuous gripper control

• mobile-base movement

• precise force regulation

• high-speed contact

• dexterous fingers

The operator is also physically interacting with the robot, which changes the system’s dynamics. The forces recorded during demonstration may include the human pushing the robot rather than forces that should occur during autonomous execution.

Kinesthetic teaching is useful for collecting motion templates and small task datasets. It is not an obvious path to internet-scale robot learning.

Press enter or click to view image in full size

3. Distributed real-robot collection

Instead of moving a robot through many environments, distributed collection sends standardized robot setups to different locations.

DROID is one of the clearest examples. It contains approximately 76,000 demonstrations, or 350 hours of interaction, collected across 564 scenes and 84 tasks by 50 collectors in North America, Asia, and Europe. The project standardized the robot hardware, sensing, teleoperation interface, and collection protocol while varying the actual environment.

This design attacks one of the biggest weaknesses of traditional lab datasets: scene diversity.

A dataset collected entirely in one laboratory may contain many trajectories, but the same tables, walls, lighting conditions, camera backgrounds, and object arrangements appear repeatedly. A model can quietly memorize those regularities.

Distributed collection introduces real variation:

• different homes

• different furniture

• different lighting

• different room layouts

• different operator habits

• different object collections

At the same time, standardized hardware prevents every data source from having a completely different action space.

Pros

• Real robot actions and real physical interaction

• Greater environmental diversity than a single laboratory

• Consistent hardware and sensor definitions

• Easier dataset aggregation

• Useful for testing out-of-distribution generalization

Cons

• Shipping, maintaining, and calibrating many systems is expensive

• Slight setup differences can still create hidden domain shifts

• Operators may interpret task instructions differently

• Data quality can vary between sites

• One standardized embodiment may limit cross-robot generality

• Troubleshooting becomes a distributed operations problem

Distributed collection is one of the strongest approaches when the objective is robust real-world behavior on a known class of robot.

Press enter or click to view image in full size

4. Federating existing robot datasets

Another strategy is to combine datasets that were originally collected by different laboratories for different purposes.

RoboNet was an early large-scale effort in this direction. It assembled 15 million video frames from seven robot platforms and showed that multi-robot pretraining could reduce the amount of data needed on a held-out robot.

Open X-Embodiment pushed the idea much further. The initial release combined more than one million real-robot trajectories from 22 embodiments, pooled from dozens of existing datasets and standardized into a common format. It included hundreds of skills and robots ranging from single arms to bimanual systems and quadrupeds.

This data has supported generalist policies such as Octo and OpenVLA. Octo was pretrained on roughly 800,000 Open X-Embodiment trajectories, while OpenVLA used approximately 970,000 episodes.

The attraction of federation

No single laboratory needs to collect everything.

Different datasets contribute different strengths:

• one contains many environments

• one contains language labels

• one contains bimanual tasks

• one contains precise assembly

• one contains mobile manipulation

• one contains unusual objects

• one contains depth or force sensing

The combined model may learn reusable visual, semantic, and motor features.

The alignment problem

Federating robot data is not equivalent to concatenating video files.

Different datasets may disagree on:

• coordinate frames

• action units

• action frequency

• image resolution

• camera count

• gripper representation

• task naming

• episode termination

• success labels

• missing observations

• controller type

Suppose one robot represents rotation using Euler-angle deltas and another uses axis-angle commands. One stores actions in the robot base frame and another uses the end-effector frame. One action means “desired pose,” while another means “change in pose during the next 100 milliseconds.”

These values cannot be mixed safely without careful normalization.

Even after standardization, the same action vector may have different physical consequences because robot dynamics, controllers, and grippers differ.

Pros

• Reuses data that already exists

• Provides cross-task and cross-embodiment diversity

• Supports foundation-model pretraining

• Encourages shared formats and open tooling

• Can improve transfer to new robots

Cons

• Inconsistent quality

• Missing or incompatible modalities

• Conflicting action semantics

• Uneven task distribution

• Duplicate or near-duplicate data

• Dataset-specific shortcuts

• Difficult attribution of which data actually helped

Federated data is excellent for broad pretraining. It is usually not a substitute for clean post-training data from the target robot.

Press enter or click to view image in full size

5. Portable proxy devices

Portable devices try to capture robot-relevant human demonstrations without operating a full robot.

The basic idea is simple: give the human a tool that resembles a robot end effector, record how that tool moves, and later transfer the trajectory to a robot.

Universal Manipulation Interface (UMI)

The Universal Manipulation Interface, or UMI, uses handheld grippers equipped with cameras and tracking. A person performs the task directly using the gripper rather than remotely controlling a robot.

UMI was designed to collect portable, in-the-wild demonstrations for dynamic, bimanual, precise, and long-horizon manipulation. Its relative-trajectory action representation and latency-matching strategy were intended to improve deployment across different robot platforms.

This changes the economics of data collection.

The operator is no longer waiting for a large robot to move. They can carry the device into a kitchen, workshop, office, or factory. Natural human motion is faster than teleoperation, and the setup is much easier to transport.

The device also constrains the demonstration toward a robot-compatible end effector. A human holding a parallel-jaw gripper cannot quietly use an unsupported finger motion that the target gripper will never reproduce.

Where proxy devices work well

• Parallel-jaw manipulation

• Bimanual tasks

• Dynamic demonstrations

• Long-horizon household activities

• Collecting across many environments

• Tasks where end-effector pose and opening width are sufficient

Where they struggle

The human arm still differs from the robot arm.

A portable gripper trajectory may pass through configurations that are easy for a person but difficult for a robot because of reach, collision, or joint-limit constraints.

The human also provides physical capabilities that the device may not measure:

• arm compliance

• body motion

• force modulation

• wrist stabilization

• subtle contact adjustments

If the gripper contains only a camera and pose tracker, the trajectory may record where it moved without recording why it succeeded.

Adding touch

FreeTacMan extends this general idea toward robot-free visuo-tactile collection. Its reported dataset contains more than three million paired visual-tactile samples and over 10,000 demonstration trajectories across 50 contact-rich tasks.

This is an important direction because it combines two forms of scalability:

1. Human-speed, robot-free demonstration

2. Direct measurement of contact-related signals

The cost is increased device complexity, calibration burden, sensor fragility, and a larger transfer problem between the handheld device and the robot end effector.

Press enter or click to view image in full size

5. Data Gloves

Data gloves measure the human hand pose using bend sensors, IMUs, optical markers, or embedded cameras. They record finger-joint motion, wrist pose, and grasp shape while a person performs a task. In this case, they sit between egocentric human video and portable proxy devices. They provide much better hand-kinematic information than video alone, but the motion still has to be retargeted to the robot hand.

Teleoperation gloves directly control a dexterous robot hand. The human finger motion is mapped to robotic finger joints, sometimes while another system controls the robot arm. These belong under direct teleoperation. Their main value is that they capture detailed dexterous actions in a robot-executable setting, although the mapping is complicated because human and robot hands have different joint structures, ranges of motion, and contact geometries.

Haptic or exoskeleton gloves add resistance or force feedback. These can let the operator feel contact, object stiffness, or grasp pressure while controlling the robot. They offer higher-fidelity teleoperation, particularly for contact-rich tasks, but they are expensive, harder to calibrate, slower to deploy, and less scalable.

Their main advantages are:

• detailed finger and wrist motion capture

• natural control of multi-finger robot hands

• less visual occlusion than camera-only hand tracking

• potential ccess to grip force and contact information

• useful data for dexterous grasping and in-hand manipulation

Their limitations are:

• calibration varies between users and hand sizes

• sensor drift and glove slippage

• human-to-robot retargeting is still required

• limited measurement of true fingertip forces unless the glove is instrumented

• reduced natural touch when the glove covers the fingertips

• higher setup time and cost than ordinary video

• difficult scaling across many operators

On the scalability versus fidelity chart, I would separate them into three positions:

• Basic motion-capture glove: medium-high scalability, medium fidelity

• Robot teleoperation glove: medium scalability, high fidelity

• Haptic exoskeleton glove: low-to-medium scalability, very high fidelity

A suitable section for the article could read:

Data gloves and haptic teleoperation

Data gloves capture the motion of the human hand using sensors embedded around the fingers and wrist. Depending on the setup, they may be used only to record human demonstrations or to directly control a dexterous robotic hand. Compared with ordinary video, gloves provide more reliable finger-joint information and are less affected by hand-object occlusion. However, the recorded motion is still human motion, not automatically robot motion. Differences in hand geometry, joint limits, and contact surfaces require retargeting before the data can be used by a robot.

More advanced systems use haptic or exoskeleton gloves that can apply resistance to the operator’s fingers. This gives the user some perception of contact while teleoperating the robot and can improve precise grasping or in-hand manipulation. The trade-off is that these systems are expensive, difficult to calibrate, and less comfortable for long collection sessions. Data gloves are therefore particularly valuable for dexterous robotic hands, but they are less necessary for robots using simple parallel-jaw grippers.

Press enter or click to view image in full size

6. Egocentric human video

Egocentric data is recorded from the human’s point of view, usually with a head-mounted camera, smart glasses, or mixed-reality headset.

This is probably the most scalable real-world source of manipulation behavior.

Humans already perform useful tasks all day. The collection device can be comparatively lightweight, and the environment does not need to be modified. A single person can generate many hours of varied interaction without a robot present.

EgoDex

EgoDex used Apple Vision Pro to collect 829 hours of egocentric manipulation video with paired 3D hand and finger tracking. It contains roughly 338,000 demonstrations across 194 tabletop tasks, totaling about 90 million frames.

This is dramatically more hand-motion data than most robot datasets can provide.

The native hand tracking is critical. Ordinary video tells us where pixels moved. Tracked skeletal data gives an estimate of:

• wrist pose

• finger-joint locations

• hand articulation

• bimanual coordination

• camera otion

More recent efforts are scaling this idea further. EgoVerse reports 1,362 hours of human demonstrations across 240 scenes, 1,965 tasks, and more than 2,000 demonstrators, with 3D hand and head poses and subtask descriptions.

EgoScale reported pretraining on more than 20,000 hours of action-labeled egocentric human video, followed by a smaller human-robot alignment stage. Its results suggest that human data can act as a reusable motor prior, particularly for dexterous manipulation, while still requiring robot-aligned data for effective transfer.

What human video gives us

Human video is excellent for learning:

• object semantics

• task intent

• scene understanding

• common action sequences

• hand-object relationships

• long-horizon task structure

• natural variation

• rare tasks

• diverse environments

• human preferences about how tasks are performed

It may teach a model that a mug is usually grasped by its body or handle, that a drawer must be opened before an object is placed inside, or that a towel should be aligned before it is folded.

What it does not give us directly

Human video usually does not provide:

• robot motor commands

• robot joint trajectories

• robot torque targets

• target-controller behavior

• robot-specific collision constraints

• reliable grip force

• full contact geometry

• exact friction

• actuator limits

• robot-camera appearance

Even perfect human hand tracking does not solve the problem.

A human hand might have more than 20 controllable degrees of freedom. A robot may have a two-finger gripper, a three-finger hand, or a dexterous hand with a different joint layout. Mapping between them is a retargeting problem.

Retargeting

Retargeting converts a human trajectory into a robot-compatible trajectory.

A simple retargeting method may minimize the distance between human and robot fingertips:

[
q_r^* =
\arg\min_{q_r}
\sum_i
|p_i^{robot}(q_r)-p_i^{human}|²
]

subject to:

• robot joint limits

• collision constraints

• contact constraints

• kinematic feasibility

But matching fingertip positions does not guarantee equivalent manipulation.

Two hands can place fingertips at the same locations while producing different:

• normal forces

• frictional forces

• palm contacts

• pressure distributions

• object stability

Retargeting also becomes ambiguous when fingers are hidden behind an object.

Object-centric alternatives

Instead of reproducing every finger joint, a system can focus on:

• object pose

• wrist pose

• contact region

• task affordance

• desired object motion

This is often more robust for robots whose hands differ substantially from human hands.

For example, if a human rotates a bottle cap, the transferable information may be:

1. stabilize the bottle

2. grasp the cap

3. apply rotation around the cap axis

4. maintain downward contact

5. stop when the thread releases

The exact human finger trajectory may matter less than the object-level constraints.

This approach has limits. In-hand rotation, finger gaiting, knot tying, and delicate tool manipulation may genuinely depend on fine finger coordination.

Pros of egocentric human data

• Very high collection speed

• Low dependence on robot availability

• Large environmental diversity

• Natural human motion

• Long-horizon task structure

• Rich semantic content

• Potentially huge numbers of operators and scenes

• Increasingly accurate hand and camera tracking

Cons

• No native robot actions

• Large embodiment gap

• Weak or absent contact measurements

• Retargeting errors

• Human hands may use capabilities unavailable to the robot

• Occlusion remains severe

• Privacy and consent issues in homes and workplaces

• Visual appearance differs from robot-mounted cameras

• Demonstrations may be physically impossible for the target robot

Human video is likely to become a major pretraining source. It is unlikely to eliminate the need for robot data.

Press enter or click to view image in full size

7. Simulation and synthetic data

Simulation offers something the real world cannot: almost unlimited resettable interaction with complete state information.

A simulator can provide:

• exact object poses

• exact contact points

• collision information

• joint states

• force values

• segmentation masks

• depth

• rewards

• success labels

• counterfactual trajectories

Objects can be randomized. Cameras can move. Lighting can change. Thousands of environments can run in parallel.

RoboCasa and RoboCasa365

RoboCasa introduced diverse simulated kitchen environments, thousands of objects, and 100 everyday tasks. It combined human demonstrations with automated trajectory generation.

RoboCasa365 expanded this into 365 tasks across 2,500 kitchen environments, with more than 600 hours of human demonstration data and over 1,600 hours of synthetically generated demonstrations.

This type of environment is useful for controlled scaling studies. Researchers can vary:

• task count

• scene count

• object diversity

• demonstration count

• robot type

and observe which form of diversity actually improves generalization.

The simulation-to-reality gap

Simulation fails when the learned policy depends on something that the simulator models incorrectly.

Common gaps include:

• friction

• deformable materials

• cable behavior

• transparent objects

• reflective surfaces

• contact compliance

• motor delays

• backlash

• camera noise

• calibration drift

• object mass

• gripper wear

A simulated cloth may be too stiff. A simulated drawer may move too smoothly. A simulated object may have an unrealistically clean collision mesh.

The policy can become excellent at exploiting these errors.

Domain randomization

Domain randomization deliberately varies simulation parameters:

• textures

• lighting

• camera pose

• object size

• friction

• mass

• latency

• noise

The idea is to prevent the model from relying on any one simulation-specific regularity.

This helps, but randomness is not automatically realism. Randomizing the wrong variables over unrealistic ranges can make training harder without improving real-world performance.

Pros

• Extremely scalable

• Parallelizable

• Safe

• Cheap resets

• Perfect labels

• Easy failure collection

• Easy rare-event generation

• Supports reinforcement learning

• Reproducible evaluation

• Can cover configurations that are dangerous to collect physically

Cons

• Imperfect physics

• Imperfect sensors

• Unrealistic object assets

• Simulated visual shortcuts

• High-quality environment creation is still expensive

• Contact-rich transfer remains difficult

• Success in simulation may say little about deployment robustness

Simulation is strongest when combined with real data, not when treated as a replacement for reality.

Press enter or click to view image in full size

8. Autonomous robot experience

Human demonstrations show what good behavior looks like. Autonomous rollouts show what the current policy actually does.

Once a robot can perform a task occasionally, it can collect additional experience by attempting the task repeatedly. This produces:

• successes

• partial successes

• failures

• recovery opportunities

• states that human demonstrators rarely visit

SOAR demonstrated autonomous real-world collection of approximately 30,500 trajectories across five tabletop environments. Its pipeline used foundation models to propose goals and estimate success, and the resulting data approximately doubled performance across its evaluation settings.

This is a potential data flywheel:

1. collect demonstrations

2. train a policy

3. deploy the policy

4. record failures and successes

5. relabel or score the experience

6. retrain

7. repeat

The difficult part is not recording the rollout. It is deciding what happened.

Autonomous labeling problems

A system must determine:

• Was the task completed?

• Was it partially completed?

• Which subtask failed?

• Did the robot damage anything?

• Is the environment safe to reset?

• Is the collected trajectory useful?

• Should the behavior be reinforced or avoided?

Incorrect success labels can poison the training data.

Resetting is another unresolved issue. A robot that places every object into a box eventually runs out of objects. Someone or something must restore the scene.

Autonomous collection is therefore easiest for tasks with automatic resets or reversible state transitions. It becomes much harder in homes and factories where actions create persistent changes.

Pros

• Collects data from the policy’s real state distribution

• Produces useful failures

• Reduces human control effort

• Supports reinforcement learning and self-improvement

• Can target uncertain or difficult states

• Becomes more valuable as the initial policy improves

Cons

• Requires a reasonably capable initial policy

• Automatic success detection is unreliable

• Resetting remains expensive

• Unsafe exploration can damage hardware or environments

• Early policies may generate repetitive, low-quality data

• Distribution can collapse around current capabilities

Autonomous data is likely to matter increasingly, but it generally sits near the top of the stack. A robot must already know enough to explore without producing only noise.

Contact data: the missing layer

Manipulation is about contact. Yet many large datasets are primarily visual.

This is understandable. Cameras are cheap, familiar, and information-dense. One image can show objects, colors, geometry, scene context, and task progress.

But images do not directly measure physical interaction.

Force-torque sensing

A six-axis force-torque sensor typically measures:

[
w = [F_x,F_y,F_z,\tau_x,\tau_y,\tau_z]
]

This describes the net force and torque passing through the sensor.

A wrist-mounted sensor can tell the robot that something is pushing against the hand. It cannot necessarily say which finger is making contact or how pressure is distributed across the gripper.

It is useful for:

• insertion

• wiping

• polishing

• tool use

• detecting unexpected collision

• estimating contact direction

• regulating applied force

Tactile sensing

Tactile sensors measure contact closer to the surface.

Depending on the technology, they may estimate:

• pressure distribution

• shear

• contact shape

• slip

• texture

• vibration

• local deformation

Tactile data is especially useful after the object becomes visually occluded by the hand.

RH20T is an early large multimodal example, with over 110,000 contact-rich sequences containing visual, force, audio, action, and, in some configurations, tactile information.

More recent datasets are becoming both larger and more specialized. OmniViTac reports more than 21,000 visuo-tactile-action trajectories across 86 tasks and over 100 objects, organized around different physical interaction patterns.

Why tactile data is difficult to scale

Tactile sensors are not passive cameras.

They touch the world, so they:

• wear

• tear

• get dirty

• deform

• drift

• requie calibration

• change the contact mechanics

• differ substantially between hardware designs

High-resolution tactile images are also expensive to store and process. A system with two tactile cameras running at high frequency can add a large amount of data to every trajectory.

Sensor placement matters. A fingertip sensor sees fingertip contact but not palm contact. A wrist force-torque sensor measures the net wrench but loses spatial detail.

There is no universally accepted tactile representation equivalent to RGB images.

Audio

Audio is often treated as an extra modality, but it can reveal important contact events:

• a latch clicking

• a gear engaging

• an object scraping

• liquid pouring

• a tool striking a surface

• a package tearing

Audio is cheap to collect and easy to synchronize. It is also affected by background noise and usually cannot localize contact by itself.

The broader lesson is that manipulation datasets should not add sensors merely because they are available. Each sensor should correspond to an uncertainty that matters for the task.

A snapshot of major datasets

The following examples illustrate how differently “large-scale” can be defined.

Press enter or click to view image in full size

A snapshot of major datasets

BridgeData V2 contains 60,096 trajectories across 24 environments. RoboMIND’s expanded release reports 107,000 demonstrations across 479 tasks, including 5,000 failure demonstrations and four robot embodiments. AgiBot World reports more than one million trajectories across 217 tasks and five deployment scenarios.

These numbers should not be treated as a leaderboard. A human-video episode, a simulated episode, and a real-robot trajectory contain different information and have different collection costs.

Data quality is an engineering problem

Once a collection method is selected, the unglamorous engineering decisions become extremely important.

A policy cannot recover information that was never recorded correctly.

Camera placement

A typical manipulation setup may contain:

• one or more static scene cameras

• a wrist camera

• a head camera

• depth sensors

Static cameras provide stable global context. They can see the table, objects, and robot configuration, but the robot may occlude the task.

Wrist cameras provide close-up views of the active area, but they suffer from:

• rapid motion

• blur

• changing viewpoint

• contact occlusion

• limited global context

Multiple views help, but they increase storage, synchronization complexity, and model cost.

Calibration

Camera intrinsics describe how 3D points project into image pixels.

Camera extrinsics describe where the camera is located relative to another coordinate frame, such as the robot base.

Incorrect calibration can make a clean-looking dataset geometrically inconsistent.

For a wrist camera, hand-eye calibration must relate the camera frame to the end-effector frame. A few millimeters of error may be acceptable for rough pick-and-place tasks and catastrophic for insertion.

Calibration should not be treated as a one-time setup step. Mounts move. Screws loosen. Robots are transported. Calibration validity should be checked throughout collection.

Synchronization

Imagine that the image at timestamp (t) is paired with the action from (t+100) milliseconds.

For slow motion, the error may appear harmless. During a fast grasp or collision, the recorded action may correspond to a completely different physical state.

Every data stream should use a clear time base:

• camera exposure time

• robot-state measurement time

• command-send time

• tactile sample time

• audio time

• operator-input time

Resampling everything to a common frequency is convenient but can hide latency.

For example, linearly interpolating joint positions may be reasonable. Interpolating a binary gripper transition or contact event may not be.

Observed state versus commanded state

The action sent to the robot is not always the action the robot executed.

A controller may saturate, clip, delay, smooth, or reject a command. The arm may collide with the environment. The gripper may stop before reaching the requested width.

A strong dataset records both:

• commanded action

• measured resulting state

Without this distinction, a model may learn from targets that were never physically realized.

Action representation

Action representation determines what the policy must learn.

Joint-space actions

Joint-space commands map directly to the robot but are highly embodiment-specific.

They can be useful when:

• the target robot is fixed

• whole-arm posture matters

• collision avoidance depends on joint configuration

• high-precision control is needed

Cartesian end-effector actions

Cartesian actions are easier to share across arms.

They can represent translation, rotation, and gripper state. An inverse-kinematics controller converts these actions into joint motion.

The drawback is that the policy loses direct control over redundant arm configurations. The same end-effector pose may correspond to several elbow positions, some safe and some unusable.

Absolute versus relative actions

Absolute actions specify a target pose.

Relative actions specify a change from the current pose.

Relative actions are often easier to normalize across workspaces and robot embodiments. Absolute actions can provide stronger global consistency when calibration is accurate.

Action chunks

Instead of predicting one action, many modern policies predict several future actions:

[
[a_t,a_{t+1},…,a_{t+H}]
]

This can produce smoother behavior and reduce the impact of inference latency.

The dataset must still support the policy’s execution frequency. Predicting a 50-step chunk from data recorded at an unstable frame rate is not a clean supervision problem.

Language labels

Language is often added after data collection.

Annotations may describe:

• the overall task

• a subtask

• the object

• the target location

• the observed result

• the reason for failure

Loose language increases semantic diversity but can introduce ambiguity.

“Put the cup away” may mean:

• place it in a cabinet

• place it in a drawer

• move it off the table

• return it to a marked location

For control, instructions should be grounded in what is visible and measurable.

Language-label generation using large models can reduce annotation cost, but generated labels should be audited. A fluent description can still be physically wrong.

Success and failure labels

Binary success labels are convenient and often inadequate.

A long task may fail after completing four of five subtasks. Treating the whole trajectory as equivalent to an immediate failure discards useful structure.

Better labels may include:

• task progress

• completed subtasks

• failure time

• failure type

• object state

• recoverability

• safety severity

RoboMIND’s inclusion of thousands of failure demonstrations is notable because failure data is still underrepresented in many large robot datasets.

Dataset splits

A random frame split is usually misleading.

Adjacent frames from one trajectory are nearly identical. If frames from the same episode appear in training and testing, evaluation becomes meaningless.

Useful test splits hold out combinations such as:

• unseen objects

• unseen object instances

• unseen rooms

• unseen camera viewpoints

• unseen operators

• unseen task formulations

• unseen robot embodiments

• unseen object-task combinations

The split should reflect the generalization claim.

A model evaluated on new positions of familiar objects is not necessarily demonstrating object generalization. A model evaluated in the same room with slightly different lighting is not demonstrating environmental generalization.

Choosing a collection method

A useful way to compare collection methods is to ask six questions.

1. Does it provide executable actions?

Direct teleoperation: yes.

Simulation: yes, within the simulated robot.

Proxy devices: approximately.

Human video: usually no.

2. Does it capture real contact dynamics?

Real-robot teleoperation: yes.

Instrumented proxy devices: partly.

Human video: indirectly.

Simulation: only as accurately as contact is modeled.

3. Can it scale across environments?

Human video: very well.

Portable devices: well.

Distributed robot collection: moderately well.

Fixed laboratory teleoperation: poorly.

Simulation: extremely well, within generated environments.

4. Can it scale across embodiments?

Human video: visually yes, mechanically no.

Federated robot data: partially.

Simulation: yes, if robot models exist.

Direct teleoperation: usually tied to the target hardware.

5. Does it include failure states?

Autonomous rollouts: naturally.

Unfiltered teleoperation: potentially.

Curated demonstrations: usually not enough.

Human video: mostly successful human behavior.

Synthetic generation: failures can be generated deliberately.

6. Is the dataset aligned with deployment?

Target-robot teleoperation has the strongest alignment.

Every step away from the target robot adds a transfer problem:

• different robot

• different controller

• different camera

• different hand

• different environment

• different physics

• different action representation

That does not make distant data useless. It changes what the data should be expected to teach.

The state of the field in 2026

The main trend is not that one data source is replacing the others.

The field is converging toward hybrid data stacks.

A modern foundation model may be trained using a combination of:

1. Internet-scale images and text for semantics

2. Human video for task structure and motion priors

3. Simulation for scale and controlled variation

4. Heterogeneous robot datasets for broad motor pretraining

5. High-quality target-robot demonstrations for alignment

6. Autonomous rollouts for failure correction and improvement

7. Tactile or force data for contact-rich post-training

GR00T N1 explicitly describes training on a heterogeneous mixture of real-robot trajectories, human videos, and synthetically generated datasets. Physical Intelligence’s (\pi_0) similarly starts from broad vision-language knowledge and diverse robot data, then uses higher-quality robot data for downstream adaptation.

This resembles a pyramid.

At the bottom is data that is abundant but weakly aligned:

• web images

• ordinary video

• egocentric human demonstrations

In the middle is data that is more structured:

• tracked hand motion

• proxy-device demonstrations

• simulated robot trajectories

• multi-embodiment robot datasets

At the top is data that is scarce but highly aligned:

• target-robot demonstrations

• real contact data

• corrections

• failures

• task-specific autonomous experience

The lower layers teach broad concepts. The upper layers teach what the particular robot must physically do.

The exact proportions depend on the task.

A robot sorting rigid packages may benefit heavily from vision and parallel-jaw teleoperation. A robot tying shoelaces will need much richer hand and contact information. A robot inserting parts on a production line may need less visual diversity but much better force control and calibration.

Press enter or click to view image in full size

Robotics data collection abundance

The open problems

We still do not know the best data mixture

More robot data helps in many settings, but different sources have different marginal value.

An additional hour of target-robot correction data may be worth more than another hundred hours of loosely related human video. At an earlier stage of training, the opposite may be true.

The field needs better methods for estimating:

• which dataset contributed to a capability

• which data is redundant

• which conditions are underrepresented

• which trajectory should be collected next

• when synthetic data stops helping

• when embodiment mismatch becomes too large

Contact remains underrepresented

Vision-centric models are improving rapidly, but many real tasks depend on physical variables that images only weakly reveal.

The open question is not whether tactile sensing is useful. It is how to collect, standardize, pretrain, and transfer tactile data at scale.

A future tactile foundation model may require data across many sensor types, contact geometries, materials, and robot hands. Today those representations remain fragmented.

Human-to-robot transfer is still incomplete

Tracked human motion is becoming abundant. Conversion into reliable robot behavior remains difficult.

The biggest gaps are:

• embodiment

• contact

• force

• occlusion

• feasibility

• camera viewpoint

• control timing

Large human datasets are most convincing as pretraining sources, not as proof that robot-specific data has become unnecessary.

Dexterous hands on a humanoid make everything harder

A parallel-jaw gripper may need one scalar to describe opening width.

A dexterous hand may require control of 15 to 25 joints, possibly with coupled or underactuated mechanisms. Small retargeting errors can cause major contact changes.

Dexterous manipulation will likely need a combination of:

• human hand data

• object tracking

• tactile sensing

• simulation

• robot-specific demonstrations

• reinforcement learning

• high-frequency local controllers

Data governance will become a real issue

Egocentric collection in homes and workplaces may capture:

• faces

• personal belongings

• confidential documents

• computer screens

• proprietary processes

• bystanders

• location information

Industrial-scale collection also raises questions about consent, worker monitoring, ownership, and compensation.

These are not separate from technical design. Privacy constraints determine where data can be collected, what can be released, and which models can be reproduced.

Evaluation is lagging behind collection

It is easier to report that a dataset contains one million trajectories than to prove that those trajectories produced a generally capable robot.

Real-world evaluation is expensive and statistically noisy. Tasks are difficult to reproduce exactly. Hardware differs. Success definitions vary.

Without stronger shared evaluation protocols, dataset scale can become a marketing number rather than a scientific measurement.

Final takeaway

The state of robotic manipulation data collection can be summarized in one sentence:

The field is getting much better at collecting different kinds of partial information.

Teleoperation gives us executable actions but is expensive.

Distributed robot collection gives us environmental diversity but creates operational overhead.

Federated datasets give us scale across robots but introduce incompatible action spaces and uneven quality.

Portable proxy devices give us faster, more natural demonstrations but still require transfer to the target robot.

Egocentric human video gives us enormous semantic and behavioral diversity but does not directly provide robot control.

Simulation gives us scale, labels, and safe exploration but models an approximation of reality.

Autonomous rollouts give us the policy’s real failures but require resets, success detection, and an already-capable system.

Tactile sensing gives us direct information about contact but remains difficult to standardize and scale.

So the winning strategy is unlikely to be one sensor, one dataset, one robot, or one collection device.

It will be a carefully constructed mixture in which each source is used for what it measures well.

Broad data teaches the model what is possible.

Robot data teaches it what is executable.

Contact data teaches it what is physically happening.

Failure data teaches it what to do when the plan stops working.

That last part may end up being the difference between a robot that performs a polished demonstration and a robot that is actually useful.

Cheers