The Waymo Open Dataset is one of the most influential public releases in autonomous driving research. When it arrived in 2019, it offered a level of sensor quality, label depth and scale that few academic teams could ever collect on their own. Since then it has grown into three separate datasets, and it remains a standard benchmark for perception and motion forecasting.
It also illustrates a broader pattern. Even the strongest public releases are built for research, from a limited set of places, under licenses that keep them out of commercial products. Understanding both sides helps teams use the data well.
According to Waymo's official overview, the Waymo Open Dataset is made up of three parts.
The Perception Dataset holds 2,030 segments of 20 seconds each, recorded at 10 Hz, or roughly 400,000 frames. It combines lidar and camera data with high-quality 3D and 2D labels, tracking IDs, segmentation labels and key points for pedestrians and cyclists.
The Motion Dataset holds 103,354 segments of 20 seconds, around 574 hours in total. These segments were mined for interesting interactions such as unprotected turns, merges and lane changes. They include object trajectories and 3D map data, which made the dataset a reference for behavior prediction research.
The End-to-End Driving Dataset, added in 2025, contains 4,021 segments with images from eight cameras that cover the full view around the vehicle. Each segment also includes routing instructions and the vehicle's past and future trajectory, which makes this part relevant to vision language action model research.
The data was collected in San Francisco, Mountain View, Los Angeles, Phoenix, Detroit and Seattle, across downtown and suburban roads, day and night, and a range of weather conditions.
Before this release, large-scale lidar data with consistent, high-quality labels was mostly locked inside companies. The release gave researchers a realistic picture of what a production-grade sensor setup sees. Its public challenges then turned 3D detection, tracking and motion forecasting into competitive, measurable problems.
The Motion Dataset was especially important. By mining real interactions instead of sampling random driving, it focused attention on the situations where prediction is hardest. Many of the methods now used to forecast how other road users will move were first proven on it. Along with nuScenes and KITTI, it became part of the shared foundation we describe in our overview of open driving datasets.
Every dataset reflects choices about where, when and why it was recorded. For the Waymo Open Dataset, four of those choices define its limits.
Six US cities. The geographic spread is wider than most research releases, yet it is still a set of American cities with American road design, signage and driving habits. A model trained mainly on this data has seen little of European roundabouts, dense two-wheeler traffic or left-hand driving.
Short, curated segments. Twenty-second segments are ideal for benchmarks. For Physical AI systems that learn how scenes evolve over minutes, such as world models, they offer only short windows into continuous driving.
Research rig, not production car. The perception data comes from Waymo's own sensor suite, with five lidars and multiple cameras. Teams building camera-first systems for consumer vehicles must adapt to a very different viewpoint and sensor mix.
Non-commercial license. Waymo states that the dataset is licensed for non-commercial use. That is appropriate for research, but it means a company cannot use the data directly to train a product it plans to sell.
There is also the question of rare events. The open release is only a small slice of the driving that a robotaxi program records. Rare situations that decide safety, from unusual road users to long-tail edge cases, appear far less often in a curated public set than they do across millions of real-world miles.
It remains one of the best benchmarks available. For comparing perception or forecasting methods against published results, it is hard to beat. Its labels are among the most carefully produced in any public release, which makes it valuable as a reference for judging your own labeling quality.
It also works well as a starting point. Teams often prototype a model on Waymo data, then fine-tune on larger licensed or internal collections that match their target regions and sensors. Cross-dataset tests, such as training on Waymo and evaluating on KITTI or nuScenes, are a quick way to see how much a model depends on one environment.
What it cannot do is replace a training set that reflects where a product will actually drive. That requires data from many more regions, collected continuously, from cameras close to production vehicles, and with enough rare events to matter. It also requires solid validation data drawn from the same real-world conditions.
For teams that need that kind of coverage, large collections of real-world driving data from everyday vehicles fill the gap that a research release was never meant to cover. When the University of Freiburg trained its Orbis 2 world model with NATIX as the largest single data contributor, it stayed robust on Waymo data it had never seen, while a model trained mainly on nuPlan and nuScenes dropped noticeably. NATIX collects synchronized multi-camera footage from factory Tesla cameras across the USA, Europe and Japan, paired with vehicle telemetry such as wheel angle, pedal position and brake state.
The Waymo Open Dataset set a high bar for what public driving data can look like. Its greatest contribution may be showing the field how much careful, large-scale data matters, and why the next step has to reach far beyond six cities.
