Multi-view three-dimensional positioning

Patent metadata

Field	Value
Publication number	US-2021183098-A1
Application number	US-202017089626-A
Country	US
Kind code	A1
Filing date	Nov 4, 2020
Priority date	Dec 13, 2019
Publication date	Jun 17, 2021
Grant date	—

Abstract

A device determines positions of objects in a scene. The device obtains object detection data (ODD) which identifies the objects and locations of reference points of the objects in 2D images of the scene. The device processes the ODD to generate candidate association data (CAD) which associates pairs of objects between the images, computes estimated 3D positions in the scene for associated pairs of objects in the CAD, and performs clustering of the estimated positions. The device further generates, based on estimated 3D positions in one or more clusters, final association data (FAD) which associates one or more objects between the images, and computes one or more final 3D positions in the scene for one or more reference points of the one or more objects in the FAD. The final 3D position(s) represent the 3D position or the 3D pose of the respective object in the scene.

First claim

1 . A method of determining positioning of objects in a scene, said method comprising:
obtaining object detection data corresponding to two-dimensional, 2D, images of the scene, said object detection data comprising an object identifier of a respective object in a respective 2D image among the 2D images of the scene and a location of a respective reference point of the respective object in the respective 2D image; processing the object detection data to generate candidate association data which associates pairs of objects between the 2D images of the scene; computing a plurality of estimated three-dimensional, 3D, positions in a scene coordinate system of the scene for associated pairs of objects in the candidate association data; determining one or more clusters of the plurality of estimated 3D positions; generating, based on estimated 3D positions in at least one cluster among the one or more clusters, final association data which associates one or more objects between the 2D images of the scene; and computing, based on the final association data, one or more final 3D positions in the scene coordinate system of the scene for one or more reference points of said one or more objects.

2 . The method of claim 1 , wherein said processing the object detection data comprises: determining a candidate correspondence of said one or more objects between pairs of 2D images among the 2D images of the scene, and wherein the candidate association data associates object identifiers between said pairs of 2D images to represent the candidate correspondence.

3 . The method of claim 2 , wherein the candidate association data further associates a set of reference points between said pairs of 2D images.

4 . The method of claim 1 , wherein said determining the one or more clusters comprises: operating a density-based clustering algorithm on the plurality of estimated 3D positions.

5 . The method of claim 1 , wherein said generating the final association data comprises: determining a primary object association between a group of 2D images for said at least one cluster, the primary object association identifying a primary object in each 2D image among the group of 2D images; computing, based on the primary object association, at least one candidate 3D position in the scene coordinate system of the scene; and projecting said at least one candidate 3D position onto the group of 2D images to generate at least one projected 2D position on said each 2D image, wherein the final association data is generated based on said at least one projected 2D position on said each 2D image.

6 . The method of claim 5 , wherein said computing the at least one candidate 3D position comprises: computing a plurality of candidate 3D positions for reference points of said primary object in said each 2D image, wherein said projecting results in projected 2D positions on said each 2D image, and wherein said generating the final association data further comprises: evaluating the projected 2D positions on said each 2D image in relation to reference points of the respective object in said each 2D image, wherein the final association data is generated based on said evaluating.

7 . The method of claim 6 , wherein said evaluating the projected 2D positions comprises: computing a comparison score for the projected 2D positions on said each 2D image in relation to the reference points of the respective object in said each 2D image; selecting, based on the comparison score, objects in the group of 2D images and including, in the final association data, an association between thus-selected objects in the group of 2D images.

8 . The method of claim 7 , wherein said generating the final association data further comprises: identifying, among the plurality of estimated 3D positions, a first set of estimated 3D positions that correspond to said association between the thus-selected objects; generating an updated plurality of estimated 3D positions by removing the first set of estimated 3D positions from the plurality of estimated 3D positions; and repeating said determining the one or more clusters and said generating the final association data for the updated plurality of estimated 3D positions.

9 . The method of claim 8 , wherein said generating the final association data further comprises: identifying a second set of estimated 3D positions which are located within a predefined distance from the one or more final 3D positions computed based on the final association data, wherein said generating the updated plurality of estimated 3D positions further comprises: removing the second set of estimated 3D positions from the plurality of estimated 3D positions.

10 . The method of claim 5 , wherein the primary object association defines a consistent association of one object between the 2D images in the group of 2D images.

11 . The method of claim 5 , wherein said determining the primary object association comprises: evaluating the estimated 3D positions in said at least one cluster to select a set of estimated 3D positions that originate from a single object in each 2D image among the group of 2D images, wherein the primary object association identifies the single object in each 2D image among the group of 2D images.

12 . The method of claim 11 , wherein said computing the plurality of estimated 3D positions comprises: assigning a score value to each estimated 3D position in the plurality of estimated 3D positions, wherein the set of estimated 3D positions is selected to optimize an aggregation of score values while ensuring that the set of estimated 3D positions originates from one object in said each 2D image in the group of 2D images.

13 . The method of claim 12 , wherein the score value is a probability value assigned by said processing the object detection data.

14 . The method of claim 1 , wherein the respective object among the associated objects in the candidate association data is assigned a plurality of reference points, and wherein said computing the plurality of estimated 3D positions is performed for a subset of the plurality of reference points of the respective object among the associated objects in the candidate association data.

15 . The method of claim 1 , wherein said computing the one or more final 3D positions comprises: combining, between the 2D images of the scene and in accordance with the final association data, locations of said one or more reference points of said one or more objects, and operating a position calculation function on the thus-combined locations to generate the one or more final positions.

16 . The method of claim 1 , wherein the respective 2D image comprises a 2D digital image, and wherein said location of the respective reference point of the respective object in the respective 2D image is given in a local coordinate system with a fixed relation to the 2D digital image.

17 . The method of claim 16 , further comprising: obtaining 2D digital images captured by imaging devices facing the scene; and processing the 2D digital images to generate said object detection data.

18 . The method of claim 1 , further comprising: matching the one or more final 3D positions to one or more final 3D positions computed at one or more preceding time points and/or at one or more succeeding time points to track said one or more objects as a function of time.

19 . A non-transitory computer-readable medium comprising computer instructions which, when executed by a processing system, cause the processing system to perform the method of claim 1 .

20 . A monito

Assignees

• Sony Corp

Inventors

• Huang Sangxia

Classifications

• G06V10/757

  Matching configurations of points or features · CPC title

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06T7/74 Primary

  involving reference images or patches · CPC title

• G06F18/23

  Clustering techniques · CPC title

• G06V20/647

  by matching two-dimensional images to three-dimensional objects · CPC title