Human Insights Driven Latent Space for Different Driving Perspectives: A Unified Encoder for Efficient Multi-Task Inference

Abstract

Autonomous driving systems require a comprehensive understanding of the environment, achieved by extracting visual features essential for perception, planning, and control. However, models trained solely on single-task objectives or generic datasets often lack the contextual information needed for robust performance in complex driving scenarios. In this work, we present a unified encoder trained across a diverse set of computer vision tasks essential for urban driving, including depth estimation, pose estimation, 3D scene flow estimation, and semantic, instance, panoptic, and motion segmentation. This single-encoder approach not only integrates these complementary visual cues, inspired by the diversity of visual cues used in human driving perception, but also enables a compact and inference-efficient model that embeds a rich, navigation-relevant latent space. Indeed, the unified encoder learns to embed multi-task knowledge into a shared representation, allowing for better downstream task adaptation, particularly for steering estimation. To ensure the efficient learning across tasks within a unified encoder, we propose a multi-scale pose decoder and employ knowledge distillation from a multi-backbone teacher model. Our experiments demonstrate that (1) the unified encoder achieves strong generalization across all visual tasks, comparable to state-of-the-art dedicated models, and (2) its frozen latent representations significantly outperform both fine-tuned models and ImageNet-pretrained baselines for steering estimation. These results underscore how multi-task feature learning, inspired by the diversity of perceptual cues used in human driving, offers an efficient and context-rich foundation for autonomous driving systems.