30 credits - LiDAR-Enhanced Vision-Language-Action Models for Safer Autonomous Trucks Introduction The VINNOVA-FFI Multi-modal Open-set Perception for Safer Autonomous Trucks MOPS project investigates how multimodal sensing and open-set perception can improve the safety of autonomous heavy-duty vehicles. Autonomous trucks must operate in open-world environments containing rare, unfamiliar, or previously unseen objects and situations. Their size, braking distance, limited maneuverability, and operational environments place particularly high demands on reliable perception and spatial reasoning. Background Camera-based end-to-end driving models capture rich semantic information, but their estimates of geometry, depth, and object distance may be sensitive to poor visibility, occlusion, and visually ambiguous scenes. LiDAR provides complementary three-dimensional measurements that could improve geometric understanding and help a driving model respond safely to unfamiliar objects and scenarios. NVIDIA Alpamayo is a family of open vision-language-action models for autonomous driving. Its models process multi-camera observations and driving context to generate trajectories together with interpretable Chain-of-Causation reasoning traces. Current open releases primarily rely on camera video and ego-motion inputs, creating an opportunity to investigate how LiDAR can be incorporated into the model’s perception, reasoning, and action-prediction pipeline. The thesis will initially rely on open-source Alpamayo models and training recipes and suitable datasets containing synchronized camera, LiDAR, and vehicle-motion data, such as the NVIDIA Physical AI Autonomous Vehicles dataset. Objective The thesis will investigate how LiDAR information can be integrated into Alpamayo to improve spatial reasoning, trajectory prediction, and robustness to unfamiliar objects and situations. Possible research directions include: representing LiDAR data as point-cloud, voxel, range-image, bird’s-eye-view, or learned tokens suitable for a vision-language-action model; comparing early, intermediate, and late fusion of camera and LiDAR features; developing token-efficient fusion mechanisms that limit the additional memory and inference cost; fine-tuning a suitable open Alpamayo release using synchronized camera and LiDAR observations; evaluating whether LiDAR improves trajectory prediction and spatial reasoning around previously unseen objects; studying whether the model’s reasoning traces correctly identify uncertainty, unfamiliar objects, and safety-relevant geometric constraints; evaluating robustness under poor visibility, occlusion, sensor degradation, or missing modalities; or analyzing which truck-relevant driving scenarios benefit most from explicit three-dimensional information. The proposed multimodal model should be compared with a camera-only Alpamayo baseline. Evaluation may include trajectory accuracy, collision-related metrics, open-set detection or response metric
