KAIST-Radar (K-Radar) (provided by 'AVELab') is a novel large-scale object detection dataset and benchmark that contains 35K frames of 4D Radar tensor (4DRT) data with power measurements along the Doppler, range, azimuth, and elevation dimensions, together with carefully annotated 3D bounding box labels of objects on the roads. K-Radar includes challenging driving conditions such as adverse weathers (fog, rain, and snow) on various road structures (urban, suburban roads, alleyways, and highways). In addition to the 4DRT, we provide auxiliary measurements from carefully calibrated high-resolution Lidars, surround stereo cameras, and RTK-GPS. This repository provides the K-Radar dataset, annotation tool for 3d bounding boxes, and the visualization tool for showing the inference results and calibrating the sensors.
The URLs listed below are useful for using the K-Radar dataset and benchmark:
- K-Radar paper and appendix
- 4D Radar object detection network of input with various density
- Enhanced network for 4D Radar using two-stage pre-processing
- Repository of Auto-labeling for 4D Radar
- Repository of Odometry for 4D Radar
- The video clip that shows each sensor measurement dynamically changing during driving under the heavy snow condition
- The video clip that shows the 4D radar tensor & Lidar point cloud (LPC) calibration and annotation process
- The video clip that shows the annotation process in the absence of LPC measurements of objects
- The video clip that shows calibration results
- The video clip that shows the GUI-based program for visualization and neural network inference
- The video clip that shows the information on tracking for multiple objects on the roads
The K-Radar dataset is published under the CC BY-NC-ND License, and all codes are published under the Apache License 2.0.
The technologies in this repository have been developed by AVELab and are being commercialized by Zeta Mobility. For commercialization inquiries, please contact Zeta Mobility (e-mail: [email protected]).
For the preparation of the dataset and pipeline, please refer to the following document: Dataset Preparation Guide.
We provide the K-Radar dataset in three ways to ensure effective deployment of the large-sized data (total 16TB):
- Access to the total dataset via our local server
- A portion of the dataset via Google Drive
- The total dataset via shipped HDD (Hard Disk Drive)
For more details, please refer to the dataset documentation.
For the preparation and quantitative results of the camera, LiDAR, and 4D Radar sensor fusion, please refer to the following document: Sensor Fusion Pipeline Guide.
The images above showcase qualitative results, with yellow boxes representing detected objects from our sensor fusion AI (camera + LiDAR + 4D Radar), which we refer to as Availability-aware Sensor Fusion (ASF). From left to right, the images depict: (1) front-facing camera images, (2) LiDAR point clouds, (3) 4D Radar tensors, and (4) sensor attention maps (SAMs). ASF stands out with three key advantages: sensor availability awareness that maintains performance even when sensors fail or degrade in adverse conditions by automatically prioritizing available sensors (in the SAMs, red, green, and blue represent attention scores for Camera, LiDAR, and 4D Radar, with predominantly blue SAMs in adverse weather indicating 4D Radar automatically received the highest attention); efficient computation structure with minimal memory usage (1.5-1.6GB) and high processing speeds (13.5-20.5Hz); and superior detection performance (87.4% AP3D at IoU=0.3 and 73.6% AP3D at IoU=0.5) even including challenging weather conditions. For more details, please refer to the paper.
For the preparation and quantitative results of the 4D Radar-based object detection, please refer to the following document: Detection Pipeline Guide.
The images above showcase qualitative results, with green boxes representing detected objects using 4D Radar. From left to right, the images depict (1) front-facing camera images (for reference), (2) LiDAR point clouds (for reference), and (3) 4D Radar tensors (input for the neural network). Note that the camera images and LiDAR point clouds are shown for reference purposes only, and the bounding boxes from the 4D Radar-only detection are projected onto these visualizations.
For the preparation and quantitative results of the 4D Radar-based object tracking, please refer to the following document: Tracking Pipeline Guide.
For the preparation and quantitative results of the pre-processing for 4D Radar, please refer to the following document: Pre-processing Pipeline Guide.
As shown in the figure above, the pre-processing of 4D Radar consists of two stages. The first stage extracts the main measurements with high power from the 4D Radar tensor (e.g., percentile in RTNH and CFAR in common), while the second stage removes noise from the first stage output. In the first stage, it is important to exclude peripheral measurements with low power and extract measurements with an appropriate density (typically 3~10%) to allow the detection network to recognize the shape of the object (refer to the paper). However, extracting measurements at this density introduces noise called sidelobe, which interferes with precise localization of the object, as shown in the figure above. Therefore, pre-processing that indicates sidelobe to the network is essential, and applying this alone greatly improves detection performance, particularly the localization performance of objects (refer to the paper). For quantitative results and pre-processed data, please refer to the pre-processing document.
For the preparation and quantitative results of the auto-labeling for 4D Radar, please refer to the following document: Auto-labeling Pipeline Guide.
The figure above shows the auto-labeling results for six road environments in the K-Radar dataset. Each cell, from left to right, displays the handmade labels, the detection results of the 4D Radar detection network trained with handmade labels (RTNH), and the network trained with automatically generated labels (i.e., auto-labels). Notably, the RTNH trained with auto-labels only from clear weather conditions performs robustly even in inclement weather. This implies that the distribution of 4D Radar data depends more on road conditions than weather conditions, allowing for the training of an all-weather resilient 4D Radar detection network using only clear weather auto-labels. For more details and quantitative results, refer to the auto-labeling document.
For the preparation and quantitative results of the 4D Radar data synthesis, please refer to the following document: Data Synthesis Pipeline Guide.
The figure above shows the synthesized 4D Radar tensor which is generated by AI, where the input is calibrated LiDAR point cloud (or camera image). We note that since 4D Radar allows lots of variations on HW design (e.g., chirp design, antenna pattern, and CFAR), it shows data with different distributions even for the same object, unlike camera and LiDAR (please refer to the 4D Radar tutorial material). Therefore, 4D Radar AI developed for specific hardware only shows reasonable performance on that hardware, and whenever the hardware changes, datasets need to be built again. Moreover, since Radar is greatly affected by the material properties of the surrounding environment (metallic or non-metallic) and multi-path effects, it is difficult to develop a realistic heuristic simulator. This 4D Radar AI synthesizer, as shown in the figure above, generates 4D Radar data that reflects the physical characteristics of 4D Radar (i.e., power and sidelobe are similar to real data). Because the synthesized 4D Radar data is physically reasonable enough, detection AI shows detection performance on real data (about 80% of real data training) even when training only with synthesized Radar data. As the Radar synthesizer improves (i.e., as synthesized data becomes more sophisticated and closer to real data), we expect that the AI synthesizer will effectively enable efficient dataset construction and can be used as a simulator.
We are targeting the energy-efficient implementation of previously GPU-developed networks (in spiking neural networks) or their implementation on economical devices (in AI accelerators). Our developed spiking neural network (which we refer to as SpikingRTNH) achieves 78% energy reduction compared to its existing GPU-based counterpart (RTNH). For more details, please refer to the paper.
For the preparation and quantitative results of the spiking neural network for 4D Radar, please refer to the following document: Spiking Neural Network Pipeline Guide.
For the preparation and quantitative results of the AI accelerator for 4D Radar, please refer to the following document: AI Accelerator Pipeline Guide.
We provide the location of a GPS antenna, essential for accurate ground-truth odometry. This location is precisely processed by integrating data from high-resolution LiDAR, RTK-GPS, and IMU data. To ensure the utmost accuracy, we verify the vehicle's location by correlating the LiDAR sensor data against a detailed, high-resolution map, as illustrated below. For security purposes, we present this location information in local coordinates rather than global coordinates (i.e., UTM). For more details, refer to the odometry document.
The K-Radar dataset is contributed by Dong-Hee Paek, Kevin Tirta Wijaya, Woo-jin Jung, Seung-Hyun Song, Hee-Yang Jung, Min-Hyeok Sun, Dong-In Kim, and Sangjae Cho, and advised by Seung-Hyun Kong.
We thank the maintainers of the following projects that enable us to develop K-Radar:
OpenPCDet by MMLAB, Rotated_IoU by lilanxiao, and kitti-object-eval-python by traveller59.
We extend our gratitude to Jen-Hao Cheng, Sheng-Yao Kuan, Aishi Huang, Hou-I Liu, Christine Wu and Wenzheng Zhao in the Information Processing Lab at the University of Washington for providing the refined K-Radar label.
This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 01210790) and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2021R1A2C3008370).
If you find this work is useful for your research, please consider citing:
@inproceedings{
paek2022kradar,
title={K-Radar: 4D Radar Object Detection for Autonomous Driving in Various Weather Conditions},
author={Dong-Hee Paek and Seung-Hyun Kong and Kevin Tirta Wijaya},
booktitle={Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2022},
url={https://openreview.net/forum?id=W_bsDmzwaZ7}
}