This repository provides instructions for downloading LiSu, a synthetic LiDAR surface normal dataset generated using CARLA. It also includes a novel method that leverages spatiotemporal information from autonomous driving data to improve LiDAR surface normal estimation, demonstrated on both LiSu and the real-world Waymo dataset.
Abstract
While surface normals are widely used to analyse 3D scene geometry, surface normal estimation from LiDAR point clouds remains severely underexplored. This is caused by the lack of large-scale annotated datasets on the one hand, and lack of methods that can robustly handle the sparse and often noisy LiDAR data in a reasonable time on the other hand. We address these limitations using a traffic simulation engine and present LiSu, the first large-scale, synthetic LiDAR point cloud dataset with ground truth surface normal annotations, eliminating the need for tedious manual labeling. Additionally, we propose a novel method that exploits the spatiotemporal characteristics of autonomous driving data to enhance surface normal estimation accuracy. By incorporating two regularization terms, we enforce spatial consistency among neighboring points and temporal smoothness across consecutive LiDAR frames. These regularizers are particularly effective in self-training settings, where they mitigate the impact of noisy pseudo-labels, enabling robust real-world deployment. We demonstrate the effectiveness of our method on LiSu, achieving state-of-the-art performance in LiDAR surface normal estimation. Moreover, we showcase its full potential in addressing the challenging task of synthetic-to-real domain adaptation, leading to improved neural surface reconstruction on real-world data.
- Python 3.10
- PyTorch 2.2.2
- CUDA 11.8
# Create virtual environment
python3.10 -m venv venv
source venv/bin/activate
# Install requirements
pip install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2 --index-url https://download.pytorch.org/whl/cu118
pip install torch-geometric -f https://data.pyg.org/whl/torch-2.2.2+cu118.html
pip install torch-scatter torch-sparse -f https://data.pyg.org/whl/torch-2.2.2+cu118.html
pip install torch-cluster -f https://data.pyg.org/whl/torch-2.2.2+cu118.html
pip install flash-attn --no-build-isolation
pip install -r requirements.txtLiSu dataset is freely accessibale on 🤗 Hugging Face:
mkdir -p data && cd data
# Make sure git-lfs is installed (https://git-lfs.com)
git lfs install
git clone https://huggingface.co/datasets/dmalic/LiSuImportant Notes:
- Storage Requirements: The dataset requires approximately 120GB of free disk space.
- Alternative Access: You can also access the dataset on Research Repository.
We support the expansion of the LiSu dataset with custom LiDAR configurations and driving routes. While our CARLA pull request is under review, you can leverage our Surface Normal LiDAR Sensor implementation from our CARLA fork.
The training procedure consists of two main stages:
- Supervised training on the synthetic LiSu dataset.
- Self-supervised domain adaptation to real-world data (e.g., Waymo, easily extendable to other datasets).
We trained our model on 8× NVIDIA A100 64GB GPUs. Below are instructions for both single-node and multi-node training.
python main.py fit -c configs/lisu_ptv3_l1+gtv+tgtv+eikonal_fp16_fulltrainval.yaml --trainer.devices 8For distributed training across multiple nodes, we provide a SLURM script example. This configures 2 nodes, each with 4 GPUs (adjustable for your cluster setup).
#!/bin/bash -l
#SBATCH -J=lisu # Job name
#SBATCH -p=<partition> # Specify cluster partition
#SBATCH -N 2 # Number of nodes
#SBATCH --ntasks-per-node=4 # Tasks (processes) per node
#SBATCH --gres=gpu:4 # GPUs per node
srun python main.py fit -c configs/lisu_ptv3_l1+gtv+tgtv+eikonal_fp16_fulltrainval.yaml \
--trainer.num_nodes 2 \ # Match SLURM node count
--trainer.devices 4 # GPUs per node (Coming soon: Instructions for adapting to real-world datasets like Waymo.)
There is a known issue with spconv regarding mixed precision evaluation (see details here). To resolve this, we have provided a patch for PointTransformerV3. You can apply the patch using:
git apply --directory=lisu/models/PointTransformerV3 ptv3.patchAll model checkpoints are hosted on 🤗 HuggingFace. To test the pretrained LiSu checkpoint, run:
python main.py test \
-c configs/sup_train/ptv3_xyz_l1+gtv+tgtv+eikonal_fp16_carla_fulltrainval.yaml \
--ckpt_path <path-to>/epoch=49-step=43600.ckpt - Release dataset
- Release CARLA surface normal LiDAR sensor
- Supervised training on LiSu dataset
- LiSu model checkpoints & evaluation
- Self-supervised domain adaptation for Waymo
- Waymo qualitative evaluation
@inproceedings{cvpr2025lisu,
title={{LiSu: A Dataset and Method for LiDAR Surface Normal Estimation}},
author={Du\v{s}an Mali\'c and Christian Fruhwirth-Reisinger and Samuel Schulter and Horst Possegger},
booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2025}
}