A Vision-Language-Action (VLA) model turns visual input and a plain language instruction into robot actions. In published stress testing, checkpoints scoring above 95% on standard benchmarks fell below 30% when a single recording condition changed. The failure is not in the architecture. It sits in the conditions your training data never varied.
The seven constants that decide if your policy transfers:
- The instruction wording
- Where the target object sat
- The camera viewpoint the scene was recorded from
- The arm’s starting joint configuration
- The room itself
- The person performing the task
- The outcome of every episode
Here is why these constants exist at all. Open X-Embodiment, the largest open robotics collection, holds roughly one million trajectories. LAION-5B holds over five billion image text pairs. Robotics is short by several orders of magnitude, so teams build volume the only way they can, by repeating the same setup. Repetition is what turns a variable into a constant.
1. The instruction wording
Most demonstration sets pair every episode with one fixed instruction string, so the model learns to ignore language entirely.
The LIBERO-Plus study out of Fudan University and Shanghai Innovation Institute built 10,030 test tasks across seven perturbation factors and 21 sub components. One test replaced the instruction with an empty value. Success on the object suite barely moved. A second test swapped the named target inside the instruction while leaving the scene identical, changing “pick up the alphabet soup” to “pick up the butter”. Success fell close to zero, and the rollouts showed the policy reaching for the original object anyway. The authors’ conclusion is blunt. The model degenerates into a vision to action mapping. The L in the acronym is decorative.
Why it matters: You are paying for language conditioning you are not receiving, and no benchmark that never rewords anything will tell you.
2. Where the target object sat
Models tolerate clutter but not movement, which means they memorised a position rather than an object.
The same team split object perturbation in two. First, add distractors drawn from a set of 416 objects. Second, move the target itself. Five models including π0, π0-Fast, UniVLA and WorldVLA barely dropped when distractors were added. The same models fell sharply once the target’s position and orientation changed, even though every semantic relation in the instruction stayed intact. The authors name this positional bias. The policy learned a coordinate, not a concept. A capture protocol that stages the same object in the same spot across thousands of episodes produces exactly this result.
Why it matters: Part of your success rate is a measurement of your staging discipline, not your model.
3. The camera viewpoint the scene was recorded from
Camera viewpoint is the single most damaging change in published testing, and it is the one most datasets never vary.
The numbers are severe. OpenVLA went from 76.5% to 1.1% when the third person camera moved. π0 went from 94.2% to 15.8%. UniVLA went from 95.2% to 4.3%. Even OpenVLA-OFT, one of the stronger baselines, gave up 37.4 points. The perturbations were not extreme. Camera distance between 1.01 and 2.00 times the original, spherical shifts inside 15 to 75 degree cones, and orientation changes as small as 2 degrees of yaw, pitch or roll. There is an encouraging half to this finding. Fine tuning on roughly 20,000 trajectories built with deliberate rig variation lifted viewpoint success from 55.6% to 92.8%.
Why it matters: If every episode you bought came from one fixed rig position, you bought one rig position.
4. The arm’s starting joint configuration
Initial joint state is the second most damaging change, and the one that added data variety helped least.
OpenVLA-OFT_m scored 97.6% clean and 30.6% once its initial joint positions were perturbed, at magnitudes between 0.1 and 0.5. π0 dropped 87.6 points on the same dimension. The authors attribute this to limited kinematic reasoning rather than a perception failure, which is a different problem with a different fix. Here is the honest part. The post trained model that lifted camera success to 92.8% reached only 30.3% on initial state, a gain of 8.6 points. Some coverage cannot be retrofitted. It has to be designed into how episodes begin, by varying the reset pose during capture instead of returning to the same home position every time.
Why it matters: Ask a vendor how many distinct starting joint configurations exist in the set before you ask how many hours it holds.
5. The room itself
Scene texture and lighting matter far less than the field assumes, and the reason is a rig detail worth knowing.
Background texture cost OpenVLA-OFT 4.7 points across a collection of 950 textures. Lighting cost 11.3. Set that against 37.4 for camera viewpoint and 59.9 for initial joint state. The explanation sits in the sensor stack. When researchers masked the third person camera entirely and left only the wrist camera, three models still scored 43.6%, 43.0% and 67.3%. Mask both views and performance collapses to near zero. The wrist camera supplies close range geometric and contact cues that illumination barely disturbs, which is why models running third person only lose over 60 points to lighting change.
Why it matters: Scene variety is the cheapest axis to add and the least decisive one to buy, unless your rig has no wrist camera at all.
6. The person performing the task
Most open sets were produced by a small number of operators repeating a narrow band of motions.
DROID, one of the larger community efforts, holds 76,000 trajectories and about 350 hours across 564 scenes and 84 tasks, produced by 50 people across 13 institutions over 12 months. HeteroMotion, published in April 2026, makes the sharper point: community datasets are dominated by structurally similar pick and place tasks, which caps coverage of motion primitives regardless of trajectory count. A 2026 data distillation result found a curated 5% subset recovering 85 to 90% of full dataset performance. Most of the volume is doing very little work.
Why it matters: Operator count and motion variety predict transfer better than trajectory count does.
7. The outcome of every episode
Almost every episode in a demonstration set ends in success, so the policy never sees a recovery it can imitate.
Collection pipelines filter for successful runs by design. The LIBERO-Plus training set itself retained only successful trajectories and stripped no op actions, which is standard practice and produces a model with no representation of what to do after a miss. Frontier recipes treat this as a first class problem rather than an edge case. The π0 authors describe their pre training mixture as deliberately built to cover recovery behaviours before any high quality post training runs. If your set contains none, your policy’s only answer to a failed grasp is to keep going.
Why it matters: A dataset of clean successes teaches your robot to be confident, not correct.
The published drop table
One finding changes how you should read the Vision-Language-Action (VLA) results below. The perturbations are not independent. Object layout alone left OpenVLA-OFT at 71.75% and camera viewpoint alone at 57.30%. Both at once produced 35.95%, lower than either factor predicts on its own. Chi square testing confirmed the interaction held across most pairs. Deployments never change one thing at a time.
All rows below are the same model, OpenVLA-OFT, tested on LIBERO-Plus.
| What changed | Success after | Points lost | What to specify instead |
| Nothing | 97.1% | 0.0 | Baseline only |
| Background texture | 92.4% | 4.7 | Low priority. Do not overpay here |
| Lighting | 85.8% | 11.3 | Vary within a scene, and check wrist coverage |
| Instruction wording | 81.5% | 15.6 | Multiple phrasings per task, plus target swaps |
| Object layout | 77.1% | 20.0 | Move targets, do not just add clutter |
| Sensor noise | 76.7% | 20.4 | Include motion blur, defocus and fog captures |
| Camera viewpoint | 59.7% | 37.4 | Multiple rig positions per task |
| Initial joint state | 37.2% | 59.9 | Distinct reset poses, planned before capture |
Read the right hand column as a shopping list. It is the difference between ordering hours and ordering coverage.
See also: Amazon Alexa Echo Uswiggerstechcrunch: The Evolution of Voice Technology
What this changes about how you buy data
No Vision-Language-Action (VLA) dataset is fixed by a bigger order of the same thing. These constants are fixed by specifying variation before capture starts, then proving it survived validation, quality control and annotation intact.
That is the argument for treating data as infrastructure rather than a delivery. Humyn Labs runs multi sensor collection in live environments, with camera, radar and IMU streams synchronised at capture rather than aligned in post, then annotated with 3D bounding boxes, point cloud segmentation, cross frame tracking and calibration validation. Every dataset clears peer review, centralised QC and a domain expert layer for safety critical work, then ships in KITTI, nuScenes, Waymo Open Dataset or custom schemas with calibration files, timestamp metadata and inter annotator agreement attached. Annotators hold verified experience in autonomy, robotics or spatial computing rather than general crowd work. You can see how the pipeline runs, or read why quality beats volume on the same data.
The same discipline applies to the two things buyers ask for after coverage. Egocentric capture, where the operator’s own view becomes the training signal rather than a fixed rig. And human in the loop correction passes, which are how you acquire the recovery episodes item 7 describes instead of hoping they appear.
Send us the ten conditions your policy has to survive. Humyn Labs scopes a collection and annotation plan within 48 hours.
Frequently asked questions
What is a Vision-Language-Action (VLA) model?
A Vision-Language-Action (VLA) model maps visual input and a natural language instruction directly to robot actions. It is built on a vision language model backbone with an action head, then trained on demonstration trajectories rather than only on text and images.
Why do VLA models ignore language instructions?
Because the instruction rarely varies during capture. When researchers removed instructions entirely from LIBERO-Plus tests, success barely changed. When they swapped the named target object, success fell close to zero. The model had learned a fixed vision to action mapping instead.
How much data do you need to train a VLA model?
There is no single number. One 2026 scaling study found success improving consistently as pre training went from 3,000 to 20,000 hours. But a curated 5% subset recovered 85 to 90% of full performance elsewhere. Coverage per condition beats raw hours.
Does more robot data always improve VLA performance?
No. A 2026 VLA scaling study, as reported in industry analysis, found that pooling heterogeneous robot datasets can induce negative transfer, meaning data from other platforms makes your model worse. Composition matters more than volume once you pass a working baseline.
What makes a good VLA training dataset?
Variation counts per condition. How many distinct camera positions, initial joint states, instruction phrasings, operators and object placements appear. Then recovery episodes rather than clean successes only, synchronised multi sensor capture, verified labels, and licensing you can defend in an audit.
How do you test a VLA model for generalization?
Change one condition at a time against a held out set, then change two at once. Published pairwise testing shows combined perturbations hurt more than either factor predicts alone. A single aggregate success rate under fixed conditions tells you almost nothing about deployment.
Before your next training run
Your Vision-Language-Action (VLA) benchmark number is measuring the conditions your data held still. Pull your capture protocol, count the distinct values behind each of the seven constants above, and you will know where the eval lift is hiding before you spend another budget cycle on hours. When you want that coverage built and verified rather than estimated, talk to Humyn Labs about physical AI data.
