Systems and methods for image reprojection

Patent metadata

Field	Value
Publication number	US-2023216999-A1
Application number	US-202217931063-A
Country	US
Kind code	A1
Filing date	Sep 9, 2022
Priority date	Dec 31, 2021
Publication date	Jul 6, 2023
Grant date	—

Abstract

An imaging system receives depth data (corresponding to an environment) from a depth sensor and first image data (a depiction of the environment) from an image sensor. The imaging system generates, based on the depth data, first motion vectors corresponding to a change in perspective of the depiction of the environment in the first image data. The imaging system generates, using grid inversion based on the first motion vectors, second motion vectors that indicate respective distances moved by respective pixels of the depiction of the environment in the first image data for the change in perspective. The imaging system generates second image data by modifying the first image data according to the second motion vectors. The second image data includes a second depiction of the environment from a different perspective than the first image data. Some image reprojection applications (e.g., frame interpolation) can be performed without the depth data.

First claim

What is claimed is:

1 . An apparatus for image processing, the apparatus comprising:
at least one memory; and at least one processor coupled to the at least one memory, the at least one processor configured to:
receive depth data including depth information corresponding to an environment;
receive first image data captured by an image sensor, the first image data including a depiction of the environment;
generate, based on at least the depth data, a first plurality of motion vectors corresponding to a change in perspective of the depiction of the environment in the first image data;
generate, using grid inversion based on the first plurality of motion vectors, a second plurality of motion vectors that indicate respective distances moved by respective pixels of the depiction of the environment in the first image data for the change in perspective;
generate second image data at least in part by modifying the first image data according to the second plurality of motion vectors, wherein the second image data includes a second depiction of the environment from a different perspective than the first image data; and
output the second image data.

2 . The apparatus of claim 1 , wherein the second image data includes an interpolated image configured to depict the environment at a second time between a first time and a third time, wherein the first image data includes at least one image depicting the environment at least at one of the first time or the third time.

3 . The apparatus of claim 1 , wherein the first image data includes a plurality of frames of video data that includes a parallax movement, wherein the second image data includes a stabilized variant of the plurality of frames of video data that reduces the parallax movement.

4 . The apparatus of claim 1 , wherein the first image data includes a person looking at the image sensor from a first angle, wherein the second image data includes the person looking at the image sensor from a second angle that is distinct from the first angle.

5 . The apparatus of claim 1 , wherein a change in perspective includes a rotation in perspective according to an angle and about an axis.

6 . The apparatus of claim 1 , wherein a change in perspective includes a translation in perspective according to a direction and a distance.

7 . The apparatus of claim 1 , wherein a change in perspective includes a transformation.

8 . The apparatus of claim 1 , wherein the change in perspective includes a movement along an axis between an original perspective of the depiction of the environment in the first image data and a position of an object in the environment, wherein at least a portion of the object is depicted in the first image data.

9 . The apparatus of claim 1 , wherein the at least one processor is configured to:
identify one or more gaps in the second image data based on one or more gaps in the second plurality of motion vectors; and modify the second image data at least in part by filling in the one or more gaps in the second image data using interpolation before outputting the second image data.

10 . The apparatus of claim 1 , wherein the at least one processor is configured to:
identify one or more occlusion areas in the second image data based on one or more gaps in the second plurality of motion vectors; and modify the second image data at least in part by filling in the one or more gaps in the second image data using inpainting before outputting the second image data.

11 . The apparatus of claim 1 , wherein the at least one processor is configured to:
identify one or more occlusion areas in the second image data based on one or more gaps in the second plurality of motion vectors; and modify the second image data at least in part by filling in the one or more gaps in the second image data using inpainting using one or more trained machine learning models before outputting the second image data.

12 . The apparatus of claim 1 , wherein the at least one processor is configured to:
identify one or more conflicts in the second image data based on one or more conflicting values from the first image data in the second plurality of motion vectors; and select one of the one or more conflicting values from the first image data based on movement data associated with the second plurality of motion vectors.

13 . The apparatus of claim 1 , wherein the depth information includes a three-dimensional representation of an environment from a first perspective.

14 . The apparatus of claim 1 , wherein the depth data is received from at least one depth sensor.

15 . The apparatus of claim 1 , further comprising:
a display, wherein, to output the second image data, the at least one processor is configured to display the second image data using at least the display.

16 . The apparatus of claim 1 , further comprising:
a communication interface, wherein, to output the second image data, the at least one processor is configured to send at least the second image data to at least a recipient device using at least the communication interface.

17 . The apparatus of claim 1 , wherein the apparatus includes at least one of a head-mounted display (HMD), a mobile handset, or a wireless communication device.

18 . A method for image processing, the method comprising:
receiving depth data including depth information corresponding to an environment; receiving first image data captured by an image sensor, the first image data including a depiction of the environment; generating, based on at least the depth data, a first plurality of motion vectors corresponding to a change in perspective of the depiction of the environment in the first image data; generating, using grid inversion based on the first plurality of motion vectors, a second plurality of motion vectors that indicate respective distances moved by respective pixels of the depiction of the environment in the first image data for the change in perspective; generating second image data at least in part by modifying the first image data according to the second plurality of motion vectors, wherein the second image data includes a second depiction of the environment from a different perspective than the first image data; and outputting the second image data.

19 . The method of claim 18 , wherein the second image data includes an interpolated image configured to depict the environment at a second time between a first time and a third time, wherein the first image data includes at least one image depicting the environment at least at one of the first time or the third time.

20 . The method of claim 18 , wherein the first image data includes a plurality of frames of video data that includes a parallax movement, wherein the second image data includes a stabilized variant of the plurality of frames of video data that reduces the parallax movement.

21 . The method of claim 18 , wherein the first image data includes a person looking at the image sensor from a first angle, wherein the second image data includes the person looking at the image sensor from a second angle that is distinct from the first angle.

22 . The method of claim 18 , wherein a change in perspective includes a rotation in perspective according to an angle and about an axis.

23 . The method of claim 18 , wherein a change in perspective includes a translation in perspective according to a direction and a distance.

24 . The method of claim 18 , wherein a c

Assignees

• Qualcomm Inc

Inventors

• Zobel Pia
• Schwartz Yuval
• Zadik Tal
• Martsiano Itschak
• Hardoon Roee
• Tzur Meir
• Gaizman Ron
• Pasternak Yehuda

Classifications

• G06T5/005

  Physics · mapped topic

• G06T2207/20081

  Training; Learning · CPC title

• H04N13/111 Primary

  Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation · CPC title

• G06T7/50

  Depth or shape recovery · CPC title

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title