Methods and systems for instance-wise segmentation of a 3D point cloud based on segmented 2D images

Patent metadata

Field	Value
Publication number	US-12518478-B2
Application number	US-202318237348-A
Country	US
Kind code	B2
Filing date	Aug 23, 2023
Priority date	Aug 23, 2023
Publication date	Jan 6, 2026
Grant date	Jan 6, 2026

Abstract

An illustrative point cloud segmentation system generates an instance-wise semantic mask for a particular source image of a set of source images that has been used to construct a 3D point cloud representing a scene that includes one or more objects. The point cloud segmentation system maps a set of 3D points from the 3D point cloud to corresponding 2D points of the particular source image, then labels, based on contours defined by the instance-wise semantic mask to demarcate the one or more objects, the mapped set of 3D points in accordance with where the corresponding 2D point for each mapped 3D point is positioned with respect to the contours. Based on the labeling of the mapped 3D points, the point cloud segmentation system produces a segmented 3D point cloud including an instance-wise segmentation of the one or more objects at the scene. Corresponding methods and systems are also disclosed.

First claim

What is claimed is:

1 . A method comprising:
generating, by a point cloud segmentation system, an instance-wise semantic mask for a particular source image of a set of source images that has been used to construct a 3D point cloud representing a scene that includes one or more objects; mapping, by the point cloud segmentation system, a set of 3D points from the 3D point cloud to corresponding 2D points of the particular source image; labeling, by the point cloud segmentation system based on contours defined by the instance-wise semantic mask to demarcate the one or more objects, the mapped set of 3D points in accordance with where the corresponding 2D point for each mapped 3D point is positioned with respect to the contours; identifying, by the point cloud segmentation system, a correlation between:
a first group of the mapped set of 3D points that are each labeled, based on the contours defined by the instance-wise semantic mask, as being associated with a first object of the one or more objects, and
a second group of 3D points mapped from the 3D point cloud to corresponding 2D points of an additional source image of the set of source images and that are each labeled, based on contours defined by an additional instance-wise semantic mask for the additional source image, as being associated with a second object of the one more objects;
determining, by the point cloud segmentation system and based on the correlation, that the first and second objects are a same object; and merging, by the point cloud segmentation system and based on the determining that the first and second objects are the same object, the first group and the second group to be labeled as being associated with the same object.

2 . The method of claim 1 , further comprising constructing, by the point cloud segmentation system and based on the set of source images, the 3D point cloud representing the scene, the constructing including determining a transformation between a 2D image space of the particular source image and a 3D world space associated with the scene;
wherein the mapping of the set of 3D points from the 3D point cloud to the corresponding 2D points of the particular source image is performed based on the transformation between the 2D image space and the 3D world space.

3 . The method of claim 1 , wherein the identifying of the correlation includes determining, during the labeling of the mapped set of 3D points, that a first mapped 3D point of the first group is already labeled as being associated with the second object.

4 . The method of claim 1 , wherein the identifying of the correlation includes identifying a geometric overlap between the first group and the second group within a 3D world space with which the 3D point cloud is associated.

5 . The method of claim 1 , wherein the mapping of the set of 3D points from the 3D point cloud to the corresponding 2D points of the particular source image includes accessing 2D-3D mapping data that is generated and stored as part of constructing the 3D point cloud based on the set of source images.

6 . The method of claim 1 , wherein the 3D point cloud is a sparse point cloud in which the set of 3D points is limited to 3D points corresponding to 2D keypoints identified in the set of source images to be associated with prominent features of the scene.

7 . The method of claim 1 , wherein the 3D point cloud is a dense point cloud and the set of 3D points includes both:
3D points corresponding to 2D keypoints identified in the set of source images to be associated with prominent features of the scene; and additional 3D points located between the 3D points corresponding to the 2D keypoints.

8 . The method of claim 1 , further comprising constructing, by the point cloud segmentation system and based on the set of source images, the 3D point cloud based on at least one of:
a multi-view stereo scene construction technique; a structure-from-motion scene construction technique; or a time-of-flight scene construction technique.

9 . The method of claim 1 , wherein
the set of source images is captured using a machine configured to gain access to an area where the scene is located.

10 . The method of claim 9 , wherein:
the scene is atop a cell tower; the one or more objects include a plurality of antennas; and the machine is a drone configured with at least one of photography or videography capabilities.

11 . The method of claim 1 , further comprising producing, by the point cloud segmentation system based on the labeling of the mapped set of 3D points, a segmented 3D point cloud including an instance-wise segmentation of the one or more objects at the scene.

12 . The method of claim 11 , further comprising using the segmented 3D point cloud to perform at least one of:
tracking an individual status for each of the one or more objects; or determining an individual physical characteristic for a particular object of the one or more objects.

13 . A system comprising:
a memory storing instructions; and one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process comprising:
generating an instance-wise semantic mask for a particular source image of a set of source images that has been used to construct a 3D point cloud representing a scene that includes one or more objects;
mapping a set of 3D points from the 3D point cloud to corresponding 2D points of the particular source image;
labeling, based on contours defined by the instance-wise semantic mask to demarcate the one or more objects, the mapped set of 3D points in accordance with where the corresponding 2D point for each mapped 3D point is positioned with respect to the contours;
identifying a correlation between:
a first group of the mapped set of 3D points that are each labeled, based on the contours defined by the instance-wise semantic mask, as being associated with a first object of the one or more objects, and
a second group of 3D points mapped from the 3D point cloud to corresponding 2D points of an additional source image of the set of source images and that are each labeled, based on contours defined by an additional instance-wise semantic mask for the additional source image, as being associated with a second object of the one more objects;

determining, based on the correlation, that the first and second objects are a same object; and
merging, based on the determining that the first and second objects are the same object, the first group and the second group to be labeled as being associated with the same object.

14 . The system of claim 13 , wherein the process further comprises constructing, based on the set of source images, the 3D point cloud representing the scene, the constructing including determining a transformation between a 2D image space of the particular source image and a 3D world space associated with the scene;
wherein the mapping of the set of 3D points from the 3D point cloud to the corresponding 2D points of the particular source image is performed based on the transformation between the 2D image space and the 3D world space.

15 . The system of claim 13 , wherein the 3D point cloud is a sparse point cloud in which the set of 3D points is limited to 3D points corresponding to 2D keypoints identified in the set of source images to be associated with prominent features of the scene.

16 . The system of claim 13 , wherein the 3D point cloud is a dense point cloud in which the set of 3D points includes both:
3D points corresponding to 2D keypoints identified in the set of source image

Assignees

• Verizon Patent & Licensing Inc

Inventors

• Murudkar Adwait Ashish
• Pantpratinidhi Nikhil Sadashiv
• Virodov Sergey
• Cole Stephen
• Seran Vidhya

Classifications

• G06V10/46

  Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features (colour feature extraction G06V10/56) · CPC title

• H04N13/351

  for displaying simultaneously · CPC title

• G06V20/70 Primary

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• G06V2201/07

  Target detection · CPC title

• G06T7/12

  Edge-based segmentation · CPC title