Introduction
LiDAR sensors are the backbone of precise 3D perception in autonomous vehicles, but they suffer from significant degradation in adverse weather (scattering in fog/rain) and are prone to hardware failures. While traditional filtering removes noise, it leaves geometric gaps that can blind downstream detectors. Conversely, 4D Imaging Radar is resilient to weather but lacks spatial resolution.
This thesis explores a novel application of Generative AI: using Latent Diffusion Models to “repair” compromised LiDAR scans. The proposed architecture utilizes the sparse but robust 4D Radar signal as a structural condition (control signal) to guide the generative process. By integrating a real-time Severity Map as an inpainting mask, the system dynamically reconstructs missing or corrupted LiDAR sectors. The result is a “Virtual Restoration” capability that recovers dense 3D geometry even when the primary sensor is partially blinded, bridging the gap between raw sensor data and robust perception.
Goals
Generative Pipeline:
- Multi-Modal Input: The architecture processes a multi-modal input consisting of the degraded LiDAR BEV features, the Spatial Severity Map (used as an inpainting mask), and the 4D Radar point cloud.
- Structural Conditioning: A dedicated encoder (e.g., PointNet++ or VoxelNet) extracts sparse geometric features from the Radar stream to serve as a physical control signal.
- Diffusion Backbone: A state-of-the-art generative network, exploring architectures such as Diffusion Transformers (DiT) or optimized U-Nets, performs the denoising process. The core mechanism utilizes Cross-Modal Attention layers to inject the Radar embeddings into the generative flow, guiding the reconstruction of the masked LiDAR sectors to align with physical reality.
Training Strategy: During training, take real/clean LiDAR scans (Ground Truth), artificially mask out sectors (simulating faults) or downsample points (simulating fog), and force the model to reconstruct the original scan using the Radar overlap.
Dataset: View-of-Delft (VoD) and MSC-RAD4R
Requirements:
- Knowledge of Python;
- Machine learning/deep learning and software development attitude (can be learned during the project);
Contact
Davide Possenti: davide.possenti@polimi.it
Stefano Arrigoni: stefano.arrigoni@polimi.it
For inquiries and further information, please email the first author, copying the other authors in CC