Method and system for rendering virtual views

Patent metadata

Field	Value
Publication number	US-9530240-B2
Application number	US-201314023168-A
Country	US
Kind code	B2
Filing date	Sep 10, 2013
Priority date	Sep 10, 2013
Publication date	Dec 27, 2016
Grant date	Dec 27, 2016

Abstract

A method including receiving a first image of a scene captured from a first perspective, the first image including an object and a background; segmenting the first image to extract a first two-dimensional contour of the object; approximating a plurality of three-dimensional locations of a plurality of points on the first contour; generating a three-dimensional billboard of the object based on the three-dimensional locations; and projecting the first image onto the three-dimensional billboard.

First claim

What is claimed is:

1. A method, comprising:
receiving a first image of a scene captured from a first perspective, the first image including an object and a background;
segmenting the first image to extract a first two-dimensional contour of the object;
receiving a second image of the scene captured from a second perspective, the second image including the object and the background;
segmenting the second image to extract a second two-dimensional contour of the object;
determining a plurality of correspondences, wherein each of the plurality of correspondences matches a first point from the first two-dimensional contour with a second point from the second two-dimensional contour;
approximating a three-dimensional contour based on the plurality of correspondences;
generating a three-dimensional billboard of the object based on interpolation of the three-dimensional contour; and
projecting at least one of the first and second image onto the three-dimensional billboard.

2. The method of claim 1 , wherein the approximating comprises determining three-dimensional locations based on the first contour and a constraint of the first contour.

3. The method of claim 2 , wherein the constraint is a placement of a first point of the first contour on a known surface.

4. The method of claim 1 , further comprising:
triangulating a best three-dimensional location for each of the plurality of correspondences to approximate the three-dimensional contour.

5. The method of claim 4 , wherein the triangulating comprises determining the best three-dimensional location as a closest point of a plurality of rays based on the first point of the correspondence and the second point of the correspondence.

6. The method of claim 1 , wherein determining the plurality of correspondences comprises using a dynamic time warping technique on the first two-dimensional contour and the second two-dimensional contour.

7. The method of claim 6 , wherein the dynamic time warping technique is a modified dynamic time warping technique.

8. The method of claim 6 , wherein the dynamic time warping technique minimizes a value of a global cost function.

9. The method of claim 8 , wherein the global cost function comprises a combination of a visual distance between two points and an epipolar distance between the two points.

10. The method of claim 1 , further comprising:
approximating a plurality of three-dimensional locations of a plurality of points on the first contour, and
filtering the plurality of three-dimensional locations to produce a filtered plurality of three-dimensional locations,
wherein the three-dimensional billboard is generated based on the filtered plurality of three-dimensional locations.

11. The method of claim 1 , further comprising:
inflating the three-dimensional billboard based on a distance function transform to produce an inflated three-dimensional billboard,
wherein the first image is projected onto the inflated three-dimensional billboard.

12. A system, comprising:
an image capture apparatus comprising a first camera, the first camera capturing a first image of a scene from a first perspective, the first image including an object and a background; and
an analysis apparatus comprising a memory including a set of instructions and a processor executing the set of instructions to perform operations comprising:
receiving the first image from the image capture apparatus;
segmenting the first image to extract a first two-dimensional contour of the object;
wherein the image capture apparatus further comprises a second camera, the second camera capturing a second image of the scene from a second perspective, the second image including the object and the background, and
wherein the operations further comprise:
receiving the second image of the scene;
segmenting the second image to extract a second two-dimensional contour of the object;
determining a plurality of correspondences, wherein each of the plurality of correspondences matches a first point from the first two-dimensional contour with a second point from the second two-dimensional contour;
approximating a three-dimensional contour based on the plurality of correspondences;
generating a three-dimensional billboard of the object based on an interpolation of the three-dimensional contour; and
projecting at least one of the first and second image onto the three-dimensional billboard.

13. The system of claim 12 , wherein the approximating comprises determining three-dimensional locations based on the first contour and a constraint of the first contour.

14. The system of claim 13 , wherein the constraint is a placement of a first point of the first contour on a known surface.

15. The system of claim 12 ,
wherein the approximating the three-dimensional locations comprises:
triangulating a best three-dimensional location for each of the plurality of correspondences to approximate the three-dimensional contour.

16. The system of claim 15 , wherein the triangulating comprises determining the best three-dimensional location as a closest point of a plurality of rays based on the first point of the correspondence and the second point of the correspondence.

17. The system of claim 12 , wherein determining the plurality of correspondences comprises using a dynamic time warping technique on the first two-dimensional contour and the second two-dimensional contour.

18. The system of claim 17 , wherein the dynamic time warping technique is a modified dynamic time warping technique.

19. The system of claim 17 , wherein the dynamic time warping technique minimizes a value of a global cost function.

20. The system of claim 19 , wherein the global cost function comprises a combination of a visual distance between two points and an epipolar distance between the two points.

21. The system of claim 12 , wherein the operations further comprise:
approximating a plurality of three-dimensional locations of a plurality of points on the first contour, and
filtering the plurality of three-dimensional locations to produce a filtered plurality of three-dimensional locations,
wherein the three-dimensional billboard is generated based on the filtered plurality of three-dimensional locations.

22. The system of claim 12 , wherein the operations further comprise:
inflating the three-dimensional billboard based on a distance function transform to produce an inflated three-dimensional billboard,
wherein the first image is projected onto the inflated three-dimensional billboard.

23. A non-transitory computer-readable storage medium storing a set of instructions executable by a processor, the set of instructions, when executed by the processor, causing the processor to perform operations comprising:
receiving a first image of a scene captured from a first perspective, the first image including an object and a background;
segmenting the first image to extract a first two-dimensional contour of the object;
receiving a second image of the scene captured from a second perspective, the second image including the object and the background;
segmenting the second image to extract a second two-dimensional contour of the object;
determining a plurality of correspondences, wherein each of the plurality of correspondences matches a first point from the first two-dimensional contour with a second point from the second two-dimensional contour;
approximating a three-dimensional contour based on the plurality of correspondences;
ge

Assignees

• Disney Entpr Inc

Inventors

• Wang Oliver
• Croci Simone
• White Marvin
• Smolic Aljoscha
• Gay Michael

Classifications

• G06T7/0083

  Physics · mapped topic

• G06T15/205 Primary

  Image-based rendering · CPC title

• G06T7/12

  Edge-based segmentation · CPC title