UBC Autonomous Systems: Real-Time Digital Twins

Why we built it

Autonomous systems generate enormous amounts of data, but without context, that data is hard to trust.

When operating UAVs and UGVs for remote sensing, diagnostics, and monitoring tasks, understanding what the robot is doing and why becomes just as important as the sensor data it collects. Logs alone were insufficient. We needed a way to observe vehicle behavior, kinematics, and decision processes in real time and retrospectively.

The core research goal was to build a digital twin platform that could mirror autonomous systems as they operated in the physical world, allowing engineers and researchers to reason about motion, sensing, and autonomy as an integrated system.

What we built

We developed a real-time digital twin framework driven by live telemetry streamed over network interfaces from UAVs and UGVs.

The digital twin acted as a live mirror of the autonomous system, enabling rapid insight into system performance and failure modes.

Key Capabilities

• Real-time vehicle rendering: Driven by streamed pose, velocity, and control state.
• Kinematic and dynamic visualization: To understand motion and actuation behavior in 3D.
• Telemetry ingestion pipelines: For position, orientation, sensor status, and decision state.
• Simulation and replay tooling: To analyze missions post-flight for root-cause analysis.
• Control system modeling: To compare expected vs. observed behavior in real-time.

Applied Research Use Cases

While these applications validated the system, the primary contribution remained the digital twin infrastructure itself.

What I learned

This work shaped how I think about autonomy and observability.

Autonomy as an Observability Problem

I learned that autonomy is not just a control problem, but an observability problem. Without clear, interpretable representations of what a system believes and how it is acting, debugging and trust break down quickly.

Physical/Digital Coupling

Building digital twins highlighted the importance of tight coupling between physical systems, telemetry, and simulation. Small discrepancies between the real world and the model often revealed deeper issues in sensing, control assumptions, or system integration.

This research laid the foundation for my later work in spatial intelligence, perception, and large-scale system monitoring.