Open driving datasets did more for autonomous driving research than almost any single algorithm. Before they existed, every lab trained and tested on its own private recordings. Results could not be compared, and progress was hard to measure. A handful of public releases changed that. They gave researchers a shared ground truth, a shared set of benchmarks and, in practice, a shared language for perception, tracking and prediction.
Those same datasets are now being asked to do something they were never designed for. World models, end-to-end driving systems and reasoning models learn from very different data than a 2012 object detector. They need long, continuous, multi-camera video from many places, recorded recently and at a scale that public releases rarely reach. This article looks at what the open datasets made possible, and at the specific places where they stop being enough.
The first contribution was comparability. When everyone evaluates on the same data, a new method either beats the previous best or it does not. That simple discipline turned autonomous driving perception into a measurable field. Leaderboards for 3D detection, tracking and motion forecasting became the place where ideas were tested in public.
The second contribution was access. Collecting a high-quality driving dataset takes a sensor-equipped vehicle, careful calibration, months of driving and expensive labeling. Universities and small teams rarely had that budget. Public driving datasets let them work on real sensor data without building a fleet, which widened the pool of people who could contribute to the field.
The third contribution was a shared vocabulary. Terms like "scene", "segment", "track" and "ego vehicle" became standard because the open datasets defined them in their formats and development kits. Tooling grew around those formats too. Many of today's perception libraries still assume a data structure that one of these releases introduced.
This matters because the field's current methods were built on that foundation. Modern camera-based 3D detection, bird's-eye-view perception and learned motion prediction all matured on open driving datasets before they reached production programs.
Label quality deserves credit as well. The teams behind these releases invested heavily in careful 3D boxes, consistent tracking IDs and clear annotation guidelines. That work set a standard that many internal labeling programs later copied. It also showed how much a model's measured performance depends on the quality of the labels it is judged against, a lesson that still shapes how teams build their own evaluation sets.
It helps to look at the major releases as generations, each one solving a limitation of the last.
The KITTI dataset was the first generation. It was released in 2012 by the Karlsruhe Institute of Technology and the Toyota Technological Institute at Chicago. It combined stereo cameras, a lidar scanner and GPS recorded around Karlsruhe, Germany, according to the KITTI benchmark site. KITTI introduced benchmarks for stereo vision, optical flow, odometry and 3D object detection, and for several years it was the default way to prove a new idea worked.
The nuScenes dataset marked the second generation. Released in 2019 by nuTonomy, now part of Motional, it offered 1,000 driving scenes of 20 seconds each from Boston and Singapore. Its key advance was full surround coverage: six cameras, a lidar and five radars, all synchronized and calibrated, adding up to about 5.5 hours of driving according to the nuScenes paper. That made it the reference for multi-camera perception and for research into how different sensors complement each other.
The Waymo Open Dataset pushed scale and label quality further. According to Waymo's dataset overview, the Perception Dataset now contains 2,030 segments of 20 seconds, and the Motion Dataset contains 103,354 segments, or 574 hours, mined for interesting interactions. A newer End-to-End Driving Dataset adds 4,021 segments with eight cameras and routing instructions. The data comes from six US cities, including San Francisco, Phoenix and Detroit.
Alongside these, BDD100K took a different path. Instead of a few hours of rich multi-sensor data, Berkeley DeepDrive collected 100,000 short dashcam videos, more than 1,000 hours in total, mostly from US cities. It showed how much variety simple, everyday recordings can capture, even from a single camera.
Motional's nuPlan, released a few years after nuScenes, moved the focus from perception to planning. It added tools for closed-loop evaluation of driving policies. That shift from "what is in the scene" to "what should the car do" foreshadowed the move toward learned driving behavior that defines the field today.
Looking across the generations, each one added a missing dimension. KITTI proved that a public benchmark could move the field. nuScenes added the full view around the vehicle. Waymo added scale and label depth, BDD100K added everyday variety, and nuPlan added the planning problem. What none of them could add was continuous growth, because a release is a snapshot by design.
Every one of these releases was a major achievement. Yet all of them share six limits that become more visible as the field moves on.
Fixed geography. Most open driving datasets come from a small number of cities. KITTI is mostly Karlsruhe, nuScenes covers parts of Boston and Singapore, and the Waymo Open Dataset spans six US cities. A model trained and tested there learns those road layouts, sign styles and driving cultures well, and it can stumble when it meets a roundabout in France or a two-wheeler-heavy street in Southeast Asia. In the Orbis 2 world model study, a model trained mainly on nuPlan and nuScenes lost noticeable accuracy on unseen Waymo data. We explored this in more depth in our article on the geography problem in autonomous driving.
Age. A dataset is a snapshot of the moment it was recorded. KITTI's cameras and vehicles date from more than a decade ago, and even the newer releases reflect roads, vehicles and sensor hardware from several years back. New vehicle designs, e-scooters, delivery robots and changing road markings all appear after the snapshot is taken, so the data slowly drifts away from the world that today's systems will drive in.
Scale. Research datasets are measured in hours. A world model or an end-to-end driving policy learns from thousands or tens of thousands of hours. The gap is large enough that public driving datasets now serve mainly as benchmarks and starting points, rather than as the main source of training data.
License. Most of the major releases are licensed for non-commercial use. The Waymo Open Dataset states this directly on its overview page, and KITTI and nuScenes use non-commercial Creative Commons terms. That is entirely reasonable for research, but it means a company building a product cannot simply train on them.
Sensor setups that do not match production cars. Many research rigs were built around expensive roof-mounted lidar and custom camera placements. Production vehicles look different. Most driver assistance systems and a growing number of autonomy programs rely mainly on cameras mounted where a carmaker can actually fit them. A model trained on a research rig's viewpoint has to be adapted before it transfers well to a production car, and that adaptation is rarely free.
Thin coverage of the long tail. Rare events are, by definition, rare. A dataset with a few thousand short segments may contain only a handful of examples of a police officer directing traffic, a vehicle reversing out of a driveway into a busy street, or heavy snow at night. That is enough to show a problem exists, but not enough to train a system to handle it reliably. The edge cases that matter most for safety are exactly the ones fixed releases capture least.
The limits above mattered less when the main task was detecting objects in single frames. They matter much more now, because the methods themselves have changed.
A World Foundation Model learns how a driving scene evolves over time. If large language models predict the next word in a sentence, world models predict the next moment in a scene. To do that well, they need long, continuous video from several viewpoints, covering many kinds of roads, weather and traffic. Short 20-second segments from a few cities give them only a narrow view of how the world behaves. We cover the broader shift in our article on why world models matter.
Reasoning systems raise the bar again. A vision language action model connects what a vehicle sees, a description of the situation and the action the driver takes. Training it requires video paired with real driving actions such as steering, braking and acceleration, along with enough variety for the model to learn why a driver behaved a certain way. Most open driving datasets record the ego vehicle's trajectory, but they were not built to capture rich driver behavior across thousands of situations.
Consistency across cameras adds another requirement. A world model that generates future frames for six or eight cameras at once has to keep every view physically consistent. A car leaving the front camera must appear correctly in the side camera a moment later. Learning that consistency takes a large volume of synchronized, calibrated multi-camera footage. Only a few open driving datasets offer full surround views at all, and none of them offers it at the scale these models now consume.
The industry has noticed this gap. NVIDIA has since released an open Physical AI autonomous vehicle dataset with 1,727 hours of driving from 25 countries. That is a significant step up from earlier public releases, and it shows where the expectation for scale and geographic spread now sits.
None of this means the public releases have lost their value. They have simply moved into a different role, and most serious teams still use them every week.
The most common use is evaluation. Because the benchmarks are fixed and widely reported, they remain the fairest way to compare a new architecture against published work. A team that claims a better 3D detector or motion forecaster is still expected to show results on nuScenes or the Waymo Open Dataset.
The second use is as a starting point. Researchers often pretrain or prototype on open driving datasets, then fine-tune on larger internal or licensed data. This lets them test an idea cheaply before committing to an expensive data program.
The third use is studying how models generalize. Because each dataset comes from different cities and sensors, training on one and testing on another is a simple way to measure how badly a model suffers from domain shift. These cross-dataset experiments have been some of the clearest evidence that geography and sensor setup matter as much as model design.
Combining datasets is harder than it sounds, though. Each release uses its own coordinate conventions, label classes, sensor layouts and file formats. Projects that merge several of them spend a large share of their effort on conversion and alignment before any training starts.
One of the quieter lessons of the open dataset era is the difference between measuring progress and producing it. A benchmark needs to be fixed, well labeled and widely used, so that results stay comparable over time. Those are exactly the properties that make it a poor long-term training source.
Training data needs the opposite qualities. It should keep growing, cover new places and conditions, and include the rare events that a fixed test set may only contain a handful of times. A model can top a leaderboard and still struggle in a city or weather condition that the benchmark never showed it.
Some benchmarks are also approaching saturation. When the top methods differ by fractions of a percentage point, the remaining gains often say more about tuning to that specific test set than about real-world ability. This is a natural stage in the life of any benchmark, and it is one reason the community keeps releasing new open driving datasets and challenges with harder scenarios, longer horizons and closed-loop evaluation.
In other words, open driving datasets remain the right tool for comparing methods. They are simply not designed to be the whole training diet for a system that must work across the world. That distinction helps explain why leading teams use public benchmarks for evaluation while sourcing most of their training data elsewhere. The alternative is a continuous data engine, which we cover in a separate article.
The next step is data that keeps pace with the world. That means continuous collection rather than one-time releases, many regions rather than a few cities, multi-camera views rather than a single forward camera, and recordings that pair video with what the driver actually did.
This is the gap NATIX was built to fill. Everyday drivers with factory Tesla cameras across the USA, Europe and Japan feed its multi-camera archive, which now holds more than 207,000 hours of video paired with vehicle telemetry and grows every month. We also publish open samples, including the 100-hour multi-camera driving dataset and a safety-critical events dataset with vehicle telemetry on Hugging Face, so researchers can test the data the same way they once tested KITTI or nuScenes.
When teams evaluate any driving dataset today, open or commercial, a few questions tend to separate useful data from merely large data:
The public benchmarks still have a role in this picture. They remain the place to compare methods fairly, while large, continuously collected autonomous driving datasets are where systems learn the variety that a fixed benchmark cannot hold. Teams evaluating that kind of data can request a sample through the same page and test it against the benchmarks they already know.
Yes, mainly for evaluation, prototyping and research into generalization. They remain the shared yardstick of the field. Their role as the main source of training data has shrunk, because modern models need far more hours, places and viewpoints than any single public release provides.
Usually not without a separate agreement. The Waymo Open Dataset, KITTI and nuScenes are all released under non-commercial terms. Teams building products should read each license carefully and plan for commercially licensed data for anything that ships.
None of the classic releases was designed for world models. Among public options, datasets with long, continuous multi-camera video and wide geographic spread are the most useful, which is why newer releases emphasize hours and countries rather than only label density. For production-scale training, most teams combine public data with larger licensed collections.
The open dataset era taught the industry how to measure progress, how to share work and how to build on each other's results. That legacy is real, and every modern driving model owes something to it.
The next era asks a different question. It is no longer enough to know which method scores best on a fixed set of scenes from a few cities. What matters now is whether a system has seen enough of the real world to behave well in places and situations no benchmark captured. Open driving datasets showed the field where to start, and the work ahead is to give models data that keeps up with how the world actually changes.
