Modeling method and apparatus using three-dimensional (3D) point cloud

Patent metadata

Field	Value
Publication number	US-10217234-B2
Application number	US-201715638518-A
Country	US
Kind code	B2
Filing date	Jun 30, 2017
Priority date	Jan 26, 2017
Publication date	Feb 26, 2019
Grant date	Feb 26, 2019

Abstract

A modeling method using a three-dimensional (3D) point cloud, the modeling method including extracting at least one region from an image captured by a camera; receiving pose information of the camera based on two-dimensional (2D) feature points extracted from the image; estimating first depth information of the image based on the at least one region and the pose information of the camera; and generating a 3D point cloud to model a map corresponding to the image based on the first depth information.

First claim

What is claimed is:

1. A modeling method using a three-dimensional (3D) point cloud, the modeling method comprising:
extracting at least one region from an image captured by a camera;
receiving pose information of the camera that is tracked based on two-dimensional (2D) feature points extracted from the image;
estimating first depth information of the image based on the at least one region and the pose information of the camera; and
generating a 3D point cloud to model a map corresponding to the image based on the first depth information.

2. The modeling method of claim 1 , wherein the pose information of the camera comprises either one or both of a key frame corresponding to the image and a pose variation that changes in response to a movement of the camera from a first position to a second position.

3. The modeling method of claim 2 , wherein the estimating comprises:
determining, based on the pose information of the camera, whether to, either one or both of, replace the key frame with a new key frame and redefine the key frame in response to the movement of the camera from the first position to the second position; and
estimating either one or both of depth information related to the new key frame and depth information related to the redefined key frame to be the first depth information of the image based on a result of the determining.

4. The modeling method of claim 3 , wherein the determining comprises:
replacing a key frame with respect to the at least one region with the new key frame in response to the pose information of the camera that changes in response to the movement of the camera being greater than or equal to a preset pose variation; and
redefining the key frame, with respect to the at least one region, in response to the pose information of the camera that changes in response to the movement of the camera being less than the preset pose variation.

5. The modeling method of claim 4 , wherein the replacing comprises replacing depth information related to the new key frame by projecting points, corresponding to the key frame, with respect to the at least one region on the new key frame.

6. The modeling method of claim 4 , wherein the redefining comprises redefining depth information related to the key frame by filtering pixels of the key frame combined with an interleaved spatial regularization through a baseline stereo comparison.

7. The modeling method of claim 1 , further comprising:
fusing the first depth information and second depth information estimated based on the 2D feature points,
wherein the generating comprises generating the 3D point cloud to modeling the map corresponding to the image based on the fused depth information.

8. The modeling method of claim 7 , wherein the fusing comprises fusing the first depth information and the second depth information by adding the second depth information to the first depth information.

9. The modeling method of claim 1 , wherein the extracting comprises searching the image for at least one region including a portion having an intensity gradient variation corresponding to a first-order differential of a pixel intensity exceeding a preset reference value.

10. The modeling method of claim 1 , further comprising:
reconstructing a 3D surface of the image based on the 3D point cloud.

11. The modeling method of claim 1 , further comprising:
extracting parameters to model a target object from the 3D point cloud; and
modeling the target object based on the parameters.

12. A non-transitory computer-readable medium storing program instructions for controlling a processor to perform the method of claim 1 .

13. A modeling method using a three-dimensional (3D) point cloud, the modeling method comprising:
detecting an edge from an image captured by a camera;
receiving information related to two-dimensional (2D) feature points extracted from the image;
analyzing a 2D contour included in the image based on the edge and the information related to the 2D feature points;
receiving pose information of the camera that is tracked based on the 2D feature points;
generating a 3D point with respect to at least a portion of the 2D contour by performing curve fitting with respect to a 3D space corresponding to the 2D contour based on a result of the analyzing and the pose information of the camera; and
generating a 3D point cloud to model a map corresponding to the image by fusing a 3D point estimated based on the 2D feature points and the 3D point generated by curve fitting.

14. The modeling method of claim 13 , wherein the analyzing comprises:
analyzing whether 2D feature points, of which 3D coordinates are verified through depth estimation, are positioned on or adjacent to a 2D contour of the edge by matching coordinates of the 2D feature points to the edge; and
determining 3D coordinates of 2D feature points positioned on, or adjacent to the 2D contour, based on the 2D feature points of which the 3D coordinates are verified.

15. The modeling method of claim 12 , wherein the generating of the 3D point comprises:
projecting 3D points of a 3D curve, predicted through the 2D contour on a 2D image, based on the pose information of the camera, an intrinsic parameter of the camera, and the 2D feature points of which the 3D coordinates are verified;
calculating a coefficient of a curve parameter of the 3D curve that minimizes an error of an error function with respect to the 3D points projected on the 2D image, the error of the error function being zero when the 3D points projected on the 2D image match the 2D contour; and
performing curve fitting with respect to the 3D space of the 2D feature points based on the coefficient of the curve parameter of the 3D curve.

16. The modeling method of claim 15 , wherein the error function is based on at least one of a 3D point corresponding to any one or any combination of any two or more of a frame of the image, the intrinsic parameter of the camera, and the pose information of the camera.

17. The modeling method of claim 13 , wherein the analyzing comprises:
obtaining a set of 2D curvatures corresponding to the 2D contour by analyzing the edge;
determining whether a curvature of the edge exceeds a preset curvature and a length of the edge exceeds a preset contour length; and
generating a contour library including any one or any combination of any two or more of coordinates of 2D feature points matched to the edge, the length of the edge, and the curvature of the edge based on a result of the determining.

18. The modeling method of claim 13 , further comprising:
reconstructing a 3D surface of the image based on the 3D point cloud.

19. The modeling method of claim 13 , further comprising:
extracting parameters to model a target object from the 3D point cloud; and
modeling the target object based on the parameters.

20. A modeling apparatus using a three-dimensional (3D) point cloud, the modeling apparatus comprising:
a communication interface configured to receive pose information of a camera that is tracked based on two-dimensional (2D) feature points extracted from an image captured by the camera; and
a processor configured:
to extract at least one region from the image,
to estimate first depth information of the image based on the at least one region and the pose information of the camera, and
to generate a 3D point cloud for modeling a map corresponding to the image based on the first depth information.

21. A modeling method using a three-dimensional (3D) point cloud,

Assignees

• Samsung Electronics Co Ltd

Inventors

• Lee Kyoobin
• Kim Seung-Chan
• Hong Sunghoon

Classifications

• G06T7/579 Primary

  from motion · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06F30/00

  Computer-aided design [CAD] · CPC title

• G06T7/73

  using feature-based methods · CPC title

• G06T7/536

  from perspective effects, e.g. by using vanishing points · CPC title