Viewpoint path stabilization

Patent metadata

Field	Value
Publication number	US-2022406003-A1
Application number	US-202117502594-A
Country	US
Kind code	A1
Filing date	Oct 15, 2021
Priority date	Jun 17, 2021
Publication date	Dec 22, 2022
Grant date	—

Abstract

Three-dimensional points may be projected onto first locations in a first image of an object captured from a first position in three-dimensional space relative to the object and projected onto second locations a virtual camera position located at a second position in three-dimensional space relative to the object. First transformations linking the first and second locations may then be determined. Second transformations transforming first coordinates for the first image to second coordinates for the second image may be determined based on the first transformations. Based on these second transformations and on the first image, a second image of the object from the virtual camera position.

First claim

1 . A method comprising:
projecting via a processor a plurality of three-dimensional points onto first locations in a first image of an object captured from a first position in three-dimensional space relative to the object; projecting via the processor the plurality of three-dimensional points onto second locations a virtual camera position located at a second position in three-dimensional space relative to the object; determining via the processor a first plurality of transformations, each of the first plurality of transformations linking a respective one of the first locations with a respective one of the second locations; based on the first plurality of transformations, determining via the processor a second plurality of transformations transforming first coordinates for the first image of the object to second coordinates for the second image of the object; and generating via the processor a second image of the object from the virtual camera position based on the first image of the object and the second plurality of transformations.

2 . The method of claim 1 , wherein the first coordinates correspond to a first-two-dimensional mesh overlain on the first image of the object, and wherein the second coordinates correspond to a second two-dimensional mesh overlain on the second image of the object.

3 . The method of claim 1 , wherein the first image of the object is one of a first plurality of images captured by a camera moving along an input path through space around the object, and wherein the second image is one of a second plurality of images generated at respective virtual camera positions relative to the object.

4 . The method of claim 3 , the method further comprising:
determining a smoothed path through space around the object based on the input path; and determining the virtual camera position based on the smoothed path.

5 . The method of claim 1 , wherein the plurality of three-dimensional points are determined at least in part via motion data captured from an inertial measurement unit at the mobile computing device.

6 . The method of claim 5 , wherein the motion data includes data selected from the group consisting of: accelerometer data, gyroscopic data, and global positioning system (GPS) data.

7 . The method of claim 1 , wherein the plurality of three-dimensional points are determined at least in part based on depth sensor data captured from a depth sensor.

8 . The method of claim 1 , wherein the second plurality of transformations is generated via a neural network.

9 . The method of claim 8 , wherein the first plurality of transformations are provided as reprojection constraints to the neural network.

10 . The method of claim 8 , wherein the neural network includes one or more similarity constraints that penalize deformation of first two-dimensional mesh via the second plurality of transformations.

11 . The method of claim 1 , the method further comprising generating a multiview interactive digital media representation (MVIDMR) that includes the second set of images, the MVIDMR being navigable in one or more dimensions

12 . The method of claim 1 , wherein the second image of the object is generated via a neural network.

13 . The method of claim 1 , wherein the processor is located within a mobile computing device that includes a camera, the first image being captured by the camera.

14 . The method of claim 1 , wherein the processor is located within a mobile computing device that includes a camera, the first image being captured by the camera.

15 . A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:
project via a processor a plurality of three-dimensional points onto first locations in a first image of an object captured from a first position in three-dimensional space relative to the object; project via the processor the plurality of three-dimensional points onto second locations a virtual camera position located at a second position in three-dimensional space relative to the object; determine via the processor a first plurality of transformations, each of the first plurality of transformations linking a respective one of the first locations with a respective one of the second locations; based on the first plurality of transformations, determine via the processor a second plurality of transformations transforming first coordinates for the first image of the object to second coordinates for the second image of the object; and generate via the processor a second image of the object from the virtual camera position based on the first image of the object and the second plurality of transformations.

16 . A computing apparatus comprising:
a processor; and a memory storing instructions that, when executed by the processor, configure the apparatus to:
project via the processor a plurality of three-dimensional points onto first locations in a first image of an object captured from a first position in three-dimensional space relative to the object;
project via the processor the plurality of three-dimensional points onto second locations a virtual camera position located at a second position in three-dimensional space relative to the object;
determine via the processor a first plurality of transformations, each of the first plurality of transformations linking a respective one of the first locations with a respective one of the second locations;
based on the first plurality of transformations, determine via the processor a second plurality of transformations transforming first coordinates for the first image of the object to second coordinates for the second image of the object; and
generate via the processor a second image of the object from the virtual camera position based on the first image of the object and the second plurality of transformations.

17 . The computing apparatus of claim 16 , wherein the first image of the object is one of a first plurality of images captured by a camera move along an input path through space around the object, and wherein the second image is one of a second plurality of images generated at respective virtual camera positions relative to the object.

18 . The computing apparatus of claim 17 , the method wherein the instructions further configure the apparatus to:
determine a smoothed path through space around the object based on the input path; and determine the virtual camera position based on the smoothed path.

19 . The computing apparatus of claim 16 , wherein the second plurality of transformations is generated via a neural network, and wherein the first plurality of transformations are provided as reprojection constraints to the neural network.

20 . The computing apparatus of claim 19 , wherein the neural network includes one or more similarity constraints that penalize deformation of first two-dimensional mesh via the second plurality of transformations.

Assignees

• Fyusion Inc

Inventors

• Cayon Rodrigo Ortiz
• Chande Krunal Ketan
• Holzer Stefan Johannes Josef
• Hwang Wook Yeon
• Trevor Alexander Jay Bruen
• Griffith Shane

Classifications

• G06T15/20 Primary

  Perspective computation · CPC title

• G06T2200/08

  involving all processing steps from image acquisition to 3D model generation · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/246

  using feature-based methods, e.g. the tracking of corners or segments · CPC title