Automatic 3D Annotation of LiDAR Data from Text Prompts

Labeling 3D boxes in LiDAR point clouds is a major cost of building perception datasets, and every new object category needs a new labeling campaign. I am developing a pipeline that produces these boxes automatically, by combining 2D vision foundation models on the camera images with the geometry of the LiDAR point cloud. No human 3D labels are used.

Teal: boxes from a LiDAR 3D detector trained only on automatically generated labels. Orange: the dataset's human annotations.

3D boxes for categories absent from the dataset's label set (street lamp, traffic sign, trash bin), generated from text prompts alone.