The editable twin becomes a labeling factory, and the resulting perception drives onboard, physics-aware tracking of cars, drones, and gates.
Onboard compute; tracks F1-tenth cars, quadrotors, and gates in two environments.
Vision-based aerial tracking is critical in GPS-denied environments. Reliable perception for tracking depends on large-scale labeled data, yet most photorealistic datasets rely on heavy manual annotation and are time-consuming to produce. We present FalconTrack, a unified perception-and-tracking framework that (i) leverages a photorealistic editable simulator for automated label generation and (ii) combines multi-head perception with physics-aware tracking for zero-shot sim-to-real transfer.
FalconTrack provides an automated labeling pipeline in a Gaussian Splatting simulator that isolates target Gaussians from short object videos and composites them with randomized backgrounds to generate RGB, mask, class, and 6-DoF pose labels, producing about 10k labeled images in under 20 minutes. Using this dataset, we train a multi-head perception module with staged learning and reprojection consistency, and connect its outputs with class-conditioned dynamics priors in an EKF for tracking.
Our perception model outperforms two baselines and reaches 96–100% class accuracy in zero-shot sim-to-real transfer on three geometrically diverse objects and two environments, while maintaining consistent performance in unseen simulated and real scenes. In real hardware closed-loop visual tracking, the onboard system runs at about 25 Hz and achieves 100% success in sim-to-real F1-tenth and gate tracking in five trajectories across two environments, while a mask-centered vision baseline drops to 60% success on F1-tenth during fast out-of-view scenarios.
Automated labeling: target Gaussians isolated from short object videos, composited with randomized backgrounds. RGB, mask, class, and 6-DoF pose labels for ~10k images in under 20 minutes.
Multi-head perception feeding class-conditioned dynamics priors in an EKF for closed-loop tracking, fully onboard at ~25 Hz.
@inproceedings{miao2026falcontrack,
title = {FalconTrack: Photorealistic Auto-Labeled Perception and Physics-Aware Vision-Based Aerial Tracking},
author = {Miao, Yan and Gandiboyina, Karteek and Giles, Noah and Okamoto, Hideki and Hoxha, Bardh and Fainekos, Georgios and Mitra, Sayan},
booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
year = {2026}
}