Scene contour recognition in video based on depth information

Patent metadata

Field	Value
Publication number	US-12456210-B2
Application number	US-202217893007-A
Country	US
Kind code	B2
Filing date	Aug 22, 2022
Priority date	Aug 31, 2020
Publication date	Oct 28, 2025
Grant date	Oct 28, 2025

Abstract

A scene contour recognition method is provided. In the method, a plurality of scene images of an environment is obtained. Three-dimensional information of a target plane in the plurality of scene images is determined based on depth information for each of the plurality of scene images, The target plane corresponds to a target object in the plurality of scene images. A three-dimensional contour corresponding to the target object is generated by fusing the target plane in each of the plurality of scene images based on the three-dimensional information of the target plane in each of the plurality of scene images. A contour diagram of the target object is generated by projecting the three-dimensional contour onto a two-dimensional plane.

First claim

What is claimed is:

1. A scene contour recognition method, the method comprising:
obtaining a plurality of scene images of an environment;
determining three-dimensional (3D) information of a target plane in the plurality of scene images based on depth information for each of the plurality of scene images, the target plane corresponding to a target object in the plurality of scene images, the 3D information of the target plane including a plane equation of the target plane and 3D coordinates corresponding to points on the target plane;
generating a 3D contour corresponding to the target object by fusing the target plane in each of the plurality of scene images based on the 3D information of the target plane in each of the plurality of scene images; and
generating a contour diagram of the target object by projecting the 3D contour onto a two-dimensional (2D) plane, wherein
the determining the 3D information of the target plane includes:
detecting the target plane in the plurality of scene images with plane fitting based on the depth information of the plurality of scene images; and
determining the 3D coordinates corresponding to the points on the target plane, and the plane equation of the target plane in a world coordinate system for the target plane in the plurality of scene images.

2. The method according to claim 1 , wherein the generating the 3D contour comprises:
determining 3D point information of edge feature points on the target plane based on the 3D information of the target plane for each of the plurality of scene images; and
generating the 3D contour corresponding to the target object by fusing the target plane in each of the plurality of scene images based on the 3D point information of the edge feature points on the target plane in each of the plurality of scene images.

3. The method according to claim 2 , wherein the generating the 3D contour corresponding to the target object comprises:
determining a target coordinate system based on a position of an image capturing apparatus at a current time, the image capturing apparatus being configured to capture the plurality of scene images;
determining target coordinates of the edge feature points in the target coordinate system according to the 3D point information of the edge feature points on the target plane, for the target plane in each of the plurality of scene images based on the target coordinate system; and
generating the 3D contour corresponding to the target object by fusing the target plane in each of the plurality of scene images based on the target coordinates of the edge feature points on the target plane in each of the plurality of scene images.

4. The method according to claim 1 , further comprising:
determining an equation corresponding to the target plane based on the 3D information of the target plane;
determining a center point of the target plane based on the equation corresponding to the target plane; and
obtaining a 3D contour of an adjusted target plane by adjusting the target plane based on a distance between an optical center of an image capturing apparatus configured to capture the plurality of scene images and the center point.

5. The method according to claim 4 , wherein the obtaining the 3D contour of the adjusted target plane comprises:
determining an optimization weight of the 3D contour based on the distance between the optical center of the image capturing apparatus and the center point, the optimization weight being inversely proportional to the distance;
determining an optimization parameter corresponding to the target plane based on the optimization weight;
adjusting the target plane based on the optimization parameter; and
obtaining the 3D contour of the adjusted target plane.

6. The method according to claim 1 , wherein the generating the contour diagram of the target object comprises:
detecting discrete points distributed on each plane in the 3D contour, and 3D coordinates corresponding to the discrete points;
generating 2D coordinates corresponding to the discrete points by performing dimension reduction on the 3D coordinates corresponding to the discrete points; and
generating the contour diagram of the target object by combining the 2D coordinates corresponding to the discrete points.

7. The method according to claim 1 , wherein after the generating the contour diagram of the target object, the method further comprises:
detecting a contour edge corresponding to the contour diagram based on discrete points in the contour diagram;
determining a contour range corresponding to the contour diagram based on the contour edge; and
obtaining an optimized contour diagram by eliminating discrete points outside the contour range based on the contour range.

8. The method according to claim 1 , wherein the obtaining the plurality of scene images comprises:
obtaining photographing parameters including a photographing period; and
capturing the plurality of scene images corresponding to each traveling position based on the photographing period in a traveling process, the plurality of scene images including a depth map and a color map.

9. The method according to claim 1 , wherein after the generating the contour diagram of the target object, the method further comprises:
determining an obtaining manner of a robotic device to obtain the target object based on a position of the target object corresponding to the contour diagram in a current traveling scenario; and
obtaining the target object based on the obtaining manner.

10. The method according to claim 9 , further comprising:
detecting a contour diagram corresponding to a target placement position of the target object;
determining a placement manner of the robotic device to place the target object based on the contour diagram corresponding to the target placement position; and
placing the target object at the target placement position based on the placement manner.

11. The method according to claim 1 , wherein after the generating the contour diagram of the target object, the method further comprises:
determining a traveling manner in a current traveling scenario based on the contour diagram, the traveling manner including at least one of a traveling direction, a traveling height, and a traveling distance; and
performing a traveling task based on the traveling manner.

12. A scene contour recognition apparatus, comprising:
processing circuitry configured to:
obtain a plurality of scene images of an environment;
determine three-dimensional (3D) information of a target plane in the plurality of scene images based on depth information for each of the plurality of scene images, the target plane corresponding to a target object in the plurality of scene images, the 3D information of the target plane including a plane equation of the target plane and 3D coordinates corresponding to points on the target plane;
generate a 3D contour corresponding to the target object by fusing the target plane in each of the plurality of scene images based on the 3D information of the target plane in each of the plurality of scene images; and
generate a contour diagram of the target object by projecting the 3D contour onto a two-dimensional (2D) plane, wherein

the determination of the 3D information of the target plane includes:
detecting the target plane in the plurality of scene images with plane fitting based on the depth information of the plurality of scene images; and
determining the 3D coordinates corresponding to the points on the target plane, and the plane equation of the target plane in a world coordinate system for the target plane in the plurality of scene images.

13. The s

Assignees

• Tencent Tech Shenzhen Co Ltd

Inventors

• Zhang Shenghao
• Ling Yonggen
• Chi Wanchao
• Zheng Yu
• Jiang Xinyang

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/10024

  Color image · CPC title

• G06V20/10 Primary

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06V10/46

  Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features (colour feature extraction G06V10/56) · CPC title