Data pours into perception systems the way rain floods a city street: streams from cameras, sensors, and roadside networks, more than any single team can neatly label. The challenge isn’t just volume; it’s bias. The most interesting moments in traffic aren’t the everyday ones that appear in textbooks, but the rare, strange, or dangerous events—the long tail of what can happen on the road. If you train a system only on the usual cases, it will stumble when the unusual occurs, and in transportation that’s precisely when safety matters most.
The Mcity Data Engine (MDE) is a bold attempt to tame that data deluge by turning unlabeled footage into a steady, iterative learning loop. Born at the University of Michigan’s Mcity, with collaborators from Karlsruhe Institute of Technology and Texas A&M University, the project is led by Daniel Bogdoll and a team of colleagues including Rajanikant Patnaik Ananta, Abeyankar Giridharan, Isabel Moore, Gregory Stevens, and Henry X. Liu. Their mission: create an open, end-to-end platform that helps researchers and practitioners identify rare and novel events directly from real-world data, label those events efficiently, and push updated perception components into the field. In other words, they’re trying to turn endless video into a smarter, safer streetside brain.
Open-Vocabulary Data Selection and the Long Tail
At the core of the Mcity approach is an idea you can imagine as a kind of linguistic radar for detection tasks. Instead of training a detector to recognize a fixed list of categories—cars, bikes, pedestrians—the system accepts a request in plain language like “pedestrian, cyclist, and other vulnerable road users” and then searches through vast, unlabeled footage to find examples that match. This is what the authors call an open-vocabulary data selection process. It relies on a robust ensemble of detectors built on four architectures, operating in tandem to cast a wide net across the unknown.
The ensemble includes four families of open-vocabulary detectors, designed to cope with the messy, real world where labels aren’t neat and tidy. By running multiple detectors in parallel and then letting their opinions collide, the system can separate the signal from the noise more reliably than any single detector could. The team then applies a consensus filter—think a majority vote among detectors—to weed out false alarms and missed detections. Only samples that meet a certain agreement are kept for labeling and retraining. This approach foregrounds the long tail: the rare, the unusual, the corner cases that don’t show up in standard datasets but show up on real streets.
Why does this matter? Because long-tail learning is where a lot of practical AI for transport stumbles. It’s easy to get good accuracy on common cases; it’s hard to generalize to the surprising ones that actually drive risk on the road. The Mcity workflow makes it feasible to hunt for those rare instances in the wild, in natural language terms, and then quickly turn the found data into better performance. It’s not magic; it’s a disciplined, scalable method for prioritizing evidence that would otherwise be drowned in the flood of generic samples. And because the system is open-source, researchers and communities can reuse, inspect, improve, and adapt the approach together rather than reinventing wheels in silos.
In practice, the data selection process scales up with hardware. The Mcity setup runs in parallel on multiple GPUs and can process large streams of footage quickly, a necessity when you’re pulling signals from millions of frames per day. The team demonstrates that their open-vocabulary ensemble can be fewer than perfect on individual detectors, but reach a robust consensus when combined, which reduces the time and labeling burden required to discover and annotate rare events. The result is a more agile, data-driven loop that mirrors the way scientists interact with a constantly changing world: propose, test on real data, refine, deploy, observe, and repeat.
The Mcity Data Engine in Practice
To turn this concept into a usable tool, the authors built a complete pipeline: data acquisition, storage, selection, labeling, model training, validation, and deployment. The data side leans on a flexible, domain-agnostic format (Voxel51) and a robust metadata store (MongoDB), letting everything from static datasets to live camera feeds flow through the same software skeleton. The practical value is clear when you consider the Smart Intersections Project in Ann Arbor, Michigan, where streams of fisheye camera data become the playground for the data engine. Here the target is rare but critical scenes involving vulnerable road users—pedestrians and cyclists who are often harder to spot in wide-angle views.
In this world, speed and accuracy are everything. The team runs a carefully curated ensemble of detectors, selecting data of interest by feeding natural-language queries like “pedestrian” or “cyclist” into the system. The detectors propose bounding boxes around candidates, and a consensus filter uses an IoU (intersection over union) threshold to reject dubious proposals. The workflow also considers the density of detections per frame; frames loaded with many potential instances of interest tend to yield more labeling value per vote. The engine is designed for real-world throughput: with 8 Nvidia H100 GPUs, it can process over a million high-resolution samples per day. The result is not just a clever idea but a scalable, practical mechanism for discovering data that would otherwise hide in the noise.
Labeling and model training follow naturally. The engine supports manual labeling via CVAT or automatic labeling from the detectors themselves, as well as from pre-trained segmentation and depth models. A key feature is how the system harmonizes different label schemes across datasets. If one dataset labels a broad class like “person” and another splits it into “pedestrian” and “cyclist”, the engine uses zero-shot similarity to map one scheme onto another, then decides whether to upscale or keep the original label. This kind of label alignment is essential when you want to merge data from diverse sources without losing semantic nuance.
The practical payoff appears in the iterative training loop. The authors describe a concrete use case with vulnerable road user detection in fisheye images. They start from a seed dataset, add newly labeled samples from one iteration of data selection (Diter), and then, in a subsequent step, incorporate crowded-scene samples (Dcrowd_iter). The impact is striking: a single iteration can boost standard evaluation metrics by a large margin while maintaining or even improving recall. In their experiments, mAP (mean average precision) at IoU 0.5 jumps, and the balance between precision and recall shifts in carefully chosen ways depending on how you weight those metrics. The upshot is that the data engine doesn’t just label more data; it guides you to label the right data, in the right way, at the right moment in the training cycle.
All of this happens in a world that matters for real deployments. The Mcity team aligns their work with actual roadside perception systems, such as Msight, to ensure that the deployed components can be integrated into edge devices or cloud back-ends. The engine’s design reflects a pragmatic philosophy: the goal is a holistic, end-to-end data-based development cycle, not a one-off algorithm or an isolated toolkit. The project’s openness—code released under MIT or similar licenses—means other researchers can iterate, test, and extend the system in ways that could accelerate progress across the field.
Why This Could Change How We Build AI Systems
The implications go beyond a single application or dataset. A public, end-to-end data engine that emphasizes long-tail discovery reframes AI development from a chase for ever-larger labeled datasets to an ongoing, data-informed dialogue with the world. By enabling natural-language queries over unlabeled footage and by leaning on a diverse ensemble of open-vocabulary detection strategies, the approach lowers the bar for teams without access to massive, curated corpora. It also lowers the risk that a model trained on one city’s roads will fail in another city with different traffic patterns, lighting, or road geometry. In short, this is a step toward a more adaptable, data-centric form of machine perception—one that learns where to look next by asking the right questions of the data itself.
Of course, there are caveats. Open-vocabulary detectors still struggle with precision in some cases, and the authors acknowledge the need for human review when novel categories emerge. The consensus filter helps, but it’s not a magic wand; it’s a statistical jury that can still be swayed by biased data or ambiguous scenes. The authors are frank about the limits and point toward future directions: expanding the vocabulary pool with more vision-language and cross-modal models, experimenting with alternative data types beyond 2D imagery, and refining the data-labeling workflow to push even higher efficiency. The hope is that, as these tools mature, researchers can spend more time ideating and less time ferrying data through ad hoc pipelines.
Viewed from a broader lens, the Mcity Data Engine is more than a technical blueprint. It’s a social technology—an open, collaborative instrument that invites researchers, practitioners, and communities to contribute, critique, and co-create. The platform’s design makes it possible to build, test, and deploy perception systems in a way that mirrors the actual process of science: formulate a hypothesis about data, collect evidence through iterative labeling and training, measure, and adjust. In an era where data becomes a strategic asset, turning it into a shared, evolving tool could accelerate progress across transportation, robotics, and beyond.
The project’s footprint extends beyond the University of Michigan. The collaboration with Karlsruhe Institute of Technology and Texas A&M grounds the work in a transatlantic, multidisciplinary ecosystem. And because the code is openly available on GitHub, the project invites the wider world to participate in the cycle of discovery, critique, and improvement. In a field where proprietary stacks often gatekeep access to data-centric development, the Mcity Data Engine stands as a vivid reminder that collaboration and openness can be engines of progress—and that the best ideas about how to learn from data benefit when more hands are on the wheel.
