Systems and methods to perform 3D localization of target objects in point cloud data using a corresponding 2D image

Patent metadata

Field	Value
Publication number	US-12437357-B2
Application number	US-202117561028-A
Country	US
Kind code	B2
Filing date	Dec 23, 2021
Priority date	Dec 23, 2021
Publication date	Oct 7, 2025
Grant date	Oct 7, 2025

Abstract

The present invention relates to a systems and methods to perform 3D localization of target objects in point cloud data using a corresponding 2D image. According to an illustrative embodiment of the present disclosure, a target environment is imaged with a camera to generate a 2D panorama and a scanner to generate a 3D point cloud. The 2D panorama is mapped to the point cloud with a 1 to 1 grid map. The target objects are detected and localized in 2D before being mapped back to the 3D point cloud.

First claim

What is claimed:

1. A computer vision system comprising:
a camera;
a light detection and ranging (LIDAR) detector; and
a processor configured to execute a plurality of machine instructions comprising:
a camera module that operates the camera to generate a 2D panoramic image;
a LIDAR module that operates the LIDAR detector to generate a 3D point cloud with distance return values;
a first mapping module that correlates the 3D point cloud to the 2D panoramic image;
a recognition module that identifies and locates at least one target object within the 2D panoramic image and generates target object data;
a second mapping module that correlates the target object data to the 2D panoramic image; and
a third mapping module that correlates the target object to the 2D panoramic image and data to the 3D point cloud;

wherein the first mapping module creates a one-to-one grid map correlating the distance return values to individuals segments of the 2D panoramic image, wherein the third mapping module uses the grid map to correlate the target object data to the 3D point cloud.

2. A method of localizing target objects in an operating environment comprising:
providing a computer vision system comprising:
a camera;
a light detection and ranging (LIDAR) detector; and
a processor configured to execute a plurality of machine instructions comprising:
a camera module that operates the camera to generate a 2D image;
a LIDAR module that operates the LIDAR detector to generate a 3D point cloud with distance return values;
a first mapping module that correlates the 3D point cloud to the 2D image;
a recognition module that identifies and locates at least one target object within the 2D image and generates target object data;
a second mapping module that correlates the target object data to the 2D image; and
a third mapping module that correlates the target object data to the 3D point cloud;

identifying a class of target objects in the operating environment;
scanning the operating environment with the scanner to generate a 3D point cloud comprising distance return values;
scanning the operating environment with the camera to generate a 2D panoramic image;
mapping the distance return values to the 2D panoramic image;
projecting the 2D panoramic image onto a cube map;
detecting the target objects within the 2D panoramic image;
generating target object data for individual boxes of the cube map; and
mapping the target object data to the 2D panoramic image and 3D point cloud.

3. The method of localizing target objects in an operating environment of claim 2 further comprising a clustering module that groups the at least one target object into at least one target group.

4. The method of localizing target objects in an operating environment of claim 2 wherein the first mapping module creates a one-to-one grid map correlating the distance return values to individual segments of the 2D panoramic image.

5. The method of localizing target objects in an operating environment of claim 2 wherein the 3D point cloud and the 2D image are comprised of unlabeled data.

6. The method of localizing target objects in an operating environment of claim 2 wherein the target objects comprise a predetermined text sequence, wherein detecting the target objects comprises searching the 2D panoramic image for text and identifying instances of the predetermined text sequence within the text.

7. The method of localizing target objects in an operating environment of claim 2 wherein the target objects comprise a predetermined text format, wherein detecting the target objects comprises searching the 2D panoramic image for text and identifying instances of the predetermined text format within the text.

8. The method of localizing target objects in an operating environment of claim 2 wherein the third mapping module uses a grid map to correlate the target object data to the 3D point cloud.

Assignees

• Naval Information Warfare Center Pacific
• Us Navy

Inventors

• Mai Adrian
• Bilinski Mark
• Provost Raymond Chey

Classifications

• G06T3/16

  Spatio-temporal transformations, e.g. video cubism · CPC title

• G06V20/63 Primary

  Scene text, e.g. street names · CPC title

• G01S17/89

  for mapping or imaging · CPC title

• G06V20/647

  by matching two-dimensional images to three-dimensional objects · CPC title

• G01S7/4808

  Evaluating distance, position or velocity data · CPC title