3d object segmentation

Patent metadata

Field	Value
Publication number	US-2016196659-A1
Application number	US-201514866771-A
Country	US
Kind code	A1
Filing date	Sep 25, 2015
Priority date	Jan 5, 2015
Publication date	Jul 7, 2016
Grant date	—

Abstract

Disclosed are a system, apparatus, and method for 3D object segmentation within an environment. Image frames are obtained from one or more depth cameras or at different times and planar segments are extracted from data obtained from the image frames. Candidate segments that comprise a non-planar object surface are identified from the extracted planar segments. In one aspect, certain extracted planar segments are identified as comprising a non-planar object surface, and are referred to as candidate segments. Confidence of preexisting candidate segments are adjusted in response to determining correspondence with a candidate segment. In one aspect, one or more preexisting candidate segments are determined to comprise a surface of a preexisting non-planar object hypothesis. Confidence in the non-planar object hypothesis is updated in response to determining correspondence with a candidate segment.

First claim

What is claimed is:

1 . A method for segmenting objects from a real-world environment, the method comprising:
extracting a first plurality of planar segments from image frame depth data; identifying, from the extracted first plurality of planar segments, one or more first candidate segments comprising a non-planar object surface; updating a confidence of a preexisting candidate segment in response to determining correspondence with the one or more first candidate segments; determining the preexisting candidate segment comprises a surface of a first preexisting non-planar object hypothesis; and updating a confidence of the first preexisting non-planar object hypothesis in response to determining correspondence with the one or more first candidate segments.

2 . The method of claim 1 , further comprising:
identifying, from the first candidate segments, one or more second candidate segments without correspondence to any preexisting candidate segments; determining correspondence between the one or more second candidate segments and a second non-planar object hypothesis; updating a confidence in the second non-planar object hypothesis in response to determining correspondence between the one or more second candidate segments and a second non-planar object hypothesis; and adding the one or more second candidate segments to a list of segments comprising the second non-planar object hypothesis.

3 . The method of claim 2 , wherein correspondence between a respective candidate segment and the preexisting non-planar object is determined according to one or more of:
a proximity between the respective candidate segment and a center of volume of the preexisting non-planar object, a volumetric intersection between a volume of the preexisting non-planar object and a volume extended from the respective candidate segment in a direction away from a camera, either: along a camera principal axis, or along a normal to the respective candidate segment, a volumetric union between two volumes, or any combination thereof.

4 . The method of claim 1 , wherein the confidence in the preexisting candidate segment is updated based on one or more of:
a number of depth frames where a surface of the preexisting candidate segment was expected to be visible, and has at least one first segment correspondence, a size of an overlap between the existing candidate segment and all the corresponding first segments, a size and shape of the one or more first corresponding segments, a number of depth frames where a preexisting candidate segment surface was expected to be visible, and there are no corresponding first segments, or any combination thereof.

5 . The method of claim 1 , wherein the confidence in the second non-planar object hypothesis is based on one or more of:
a number of candidate segments currently listed as comprising a surface of the second non-planar object hypothesis, confidence in candidate segments currently listed as comprising a surface of the second non-planar object hypothesis, a size of overlap of the volumes in space defined by the candidate segments exceeding a threshold of confidence, object volume size, proximity of the non-planar object hypothesis to other geometric primitives extracted from the environment, wherein the geometric primitives are one or more of: other objects, or larger planar segments.

6 . The method of claim 1 , further comprising:
creating a new non-planar object hypothesis comprising one or more third candidate segments having no correspondence to either: previously determined candidate segments, or preexisting non-planar object hypotheses.

7 . The method of claim 1 , wherein identifying a respective candidate segment comprises determining one or more of:
whether an absolute size of the respective candidate segment is larger than a first threshold and smaller than a second threshold, whether a ratio between a first and second principal component of a point cloud associated with the respective segment is smaller than a third threshold, whether an angle between a camera principal axis and a segment normal is greater than fourth threshold, or any combination thereof.

8 . The method of claim 1 , wherein correspondence between two or more candidate segments is according to one or more of:
a distance between planar segments or convex hulls of the two or more candidate segments to compare, an angle between corresponding planes of the two or more candidate segments to compare, an overlap measure between a first segment from the two or more candidate segments and a projection of the first segment onto a plane defined by a second segment from the two or more candidate segments, or any combination thereof.

9 . The method of claim 1 , further comprising:
providing, in response to determining the confidence in the second non-planar object hypothesis meets a threshold object confidence, one or both of: a representation of a 3D boundary encompassing an object volume for the second non-planar object hypothesis, or volume occupancy information of the second non-planar object hypothesis.

10 . The method of claim 1 , further comprising:
removing the preexisting candidate segment from a list of preexisting segments when confidence in the preexisting candidate segment drops below a threshold; and removing the preexisting non-planar object hypothesis when confidence in the preexisting non-planar object hypothesis drops below a threshold.

11 . The method of claim 1 , further comprising:
detecting a change in a number or confidence of candidate segments comprising a surface of the preexisting non-planar object hypothesis; and updating, in response to detecting the change, volume occupancy properties associated with the preexisting non-planar object hypothesis.

12 . A machine readable non-transitory storage medium having stored therein program instructions that are executable by a processor to:
extract a first plurality of planar segments from image frame depth data; identify, from the extracted first plurality of planar segments, one or more first candidate segments comprising a non-planar object surface; update a confidence of a preexisting candidate segment in response to determining correspondence with the one or more first candidate segments; determine the preexisting candidate segment comprises a surface of a first preexisting non-planar object hypothesis; and update a confidence of the first preexisting non-planar object hypothesis in response to determining correspondence with the one or more first candidate segments.

13 . The medium of claim 12 , further comprising instructions to:
identify, from the first candidate segments, one or more second candidate segments without correspondence to any preexisting candidate segments; determine correspondence between the one or more second candidate segments and a second non-planar object hypothesis; update a confidence in the second non-planar object hypothesis in response to determining correspondence between the one or more second candidate segments and a second non-planar object hypothesis; and add the one or more second candidate segments to a list of segments comprising the second non-planar object hypothesis.

14 . The medium of claim 13 , wherein correspondence between a respective candidate segment and the preexisting non-planar object is determined according to one or more of:
a proximity between the respective candidate segment and a center of volume of the preexisting non-planar object, a volumetric intersection between a volume of the preexisting non-planar object and a volume extended fro

Assignees

• Qualcomm Inc

Inventors

• Vrcelj Bojan
• Lee Young Hoon
• Reitmayr Gerhard

Classifications

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06T7/0081 Primary

  Physics · mapped topic

• G06K9/6215

  Physics · mapped topic

• G06T2207/10012

  Stereo images · CPC title

• H04N13/0203

  Electricity · mapped topic