The nuScenes dataset changed what researchers expected a driving dataset to include. Released in 2019 by nuTonomy, now part of Motional, it was the first major public release to capture the full view around a vehicle with cameras, lidar and radar together. That made it the reference point for 360-degree perception, and it still anchors many of the benchmarks used to compare camera-based 3D detection today.
Its design also shows the limits of any fixed research release. A dataset recorded in two cities over a short period can teach the field a great deal, but it cannot keep up with the scale and variety that modern models now need.
According to the nuScenes website, the dataset contains 1,000 driving scenes of 20 seconds each, recorded in Boston and Singapore. Each scene comes from a car equipped with six cameras, one lidar and five radars, all synchronized and calibrated. Together the scenes add up to about 5.5 hours of driving.
The scenes were chosen to include interesting and difficult situations, such as dense traffic, construction and night driving. Objects are labeled with 3D bounding boxes, attributes and tracking IDs across 23 classes. The dataset is released under a non-commercial Creative Commons license, with a separate route for commercial licensing.
Before nuScenes, most public data focused on what was in front of the car. Real driving does not work that way. A lane change depends on what is behind and beside you, and an intersection needs awareness in every direction.
By providing six cameras with overlapping fields of view, the nuScenes dataset made surround-view perception a mainstream research problem. Bird's-eye-view methods, which fuse multiple camera views into one top-down picture of the scene, matured largely on nuScenes. The radar data also enabled early research into how different sensors complement each other in rain and darkness.
That surround-view idea is now central to modern autonomy. Open-source driving stacks such as Autoware have moved toward multi-camera data for end-to-end driving, and world models are increasingly trained on synchronized multi-view video.
The nuScenes dataset came with its own evaluation metrics, designed for full 3D scenes rather than single images. The headline metric, the nuScenes Detection Score described in the original nuScenes paper, combines detection accuracy with how well a model estimates an object's position, size, orientation, speed and attributes. That pushed researchers to care about the full description of each object, not just whether a box was drawn in roughly the right place.
The dataset also supported tracking, prediction and segmentation tasks, along with a lidar segmentation extension called nuScenes-lidarseg. Its development kit, often called the nuScenes devkit, provides tools for loading data, visualizing scenes and running the official evaluation. Clear tools like these are one reason the dataset spread so quickly through research labs.
Because so many papers report results on the same nuScenes benchmarks, the numbers have become a common language for camera-based 3D perception. When a new bird's-eye-view method claims progress, the nuScenes leaderboard is usually where that claim is tested first.
Motional followed nuScenes with nuPlan, a dataset focused on planning rather than perception. It offered around 1,200 hours of driving from Boston, Pittsburgh, Las Vegas and Singapore, with tools for closed-loop simulation of driving policies.
That shift matters. nuScenes asked "what is in the scene?" while nuPlan asked "what should the car do next?" The second question is closer to how today's end-to-end and reasoning systems, including the vision language action model, approach driving.
For all its influence, the nuScenes dataset has clear limits for modern training.
Two cities. Boston and Singapore are very different from each other, which is useful, but they are still only two cities. Models trained on them see a narrow slice of road layouts, signage and driving cultures.
Small scale. Those 1,000 scenes total about 5.5 hours of driving. That is plenty for a benchmark, but tiny compared with the thousands of hours that World Foundation Models and end-to-end policies learn from. Even nuPlan's 1,200 hours sit well below that bar.
A research rig. The sensor layout, with lidar and five radars, reflects a robotaxi research vehicle rather than the camera setups of consumer cars.
Non-commercial terms. Like most open releases, it cannot be used directly to train commercial products without a separate license.
The nuScenes dataset remains one of the most useful benchmarks in the field. Its development kit, clear formats and active leaderboards make it the default way to report results for camera-based 3D detection and tracking. Many teams also use it to prototype multi-camera models before scaling up.
What it cannot provide is the volume and geographic spread needed for production training. Research on world model scaling laws points in the same direction: performance keeps improving with more diverse, multi-camera data, well beyond the size of any single benchmark.
That is where continuously collected, multi-camera driving dataset options come in. Valeo's VATIX world model, trained on 5,500 hours of NATIX footage from 28 countries, was evaluated on nuScenes without ever training on it and still performed strongly. NATIX gathers synchronized multi-camera footage from everyday Tesla drivers across the USA, Europe and Japan, paired with vehicle telemetry such as wheel angle, pedal position and brake state.
The nuScenes dataset taught the field to look all the way around the vehicle. The next lesson is to look across many more cities and many more hours than any fixed release can hold. We bring these lessons together in our review of the limits of open datasets.
