Adaptive parameters in image regions based on eye tracking information

Patent metadata

Field	Value
Publication number	US-10373592-B2
Application number	US-201715662582-A
Country	US
Kind code	B2
Filing date	Jul 28, 2017
Priority date	Aug 1, 2016
Publication date	Aug 6, 2019
Grant date	Aug 6, 2019

Abstract

A display system divides a screen into regions and applies a different set of rendering/encoding parameters to each region. The system applies a first set of parameters to a first region that is being viewed by a fovea of an eye of a user. The system may also apply a second set of parameters to a second region that is being viewed by a parafovea of the eye, and apply a third set of parameters to a third region that is being viewed by the area of the eye outside of the parafovea. The first set of parameters are selected to yield relatively high image quality, while the second set of parameters are yield intermediate quality, and the third set of parameters yield lower quality. As a result, the second region and the third region can be rendered, encoded, and transmitted with less computing power and less bandwidth.

First claim

What is claimed is:

1. A method comprising:
receiving eye tracking information comprising at least one image of an eye of a user viewing a screen of a display device;
determining, based on the eye tracking information, a first screen region corresponding to a region of the screen, the first screen region containing a point of regard of the user, the point of regard representing a point on the screen at which the user is looking at the time the eye tracking information was captured;
determining, based on the eye tracking information, a second screen region corresponding to a region of the screen that is separate from the first screen region;
determining, based on the eye tracking information, a user's depth of focus;
encoding an image for display on the screen, the image comprising an object with at least, a portion of the object to be displayed within the first screen region, the encoding comprising:
encoding a first image region of the image based on a first set of encoding parameters, the first image region comprising a portion of the image to be displayed in the first screen region,
encoding a second image region of the image based on a second set of encoding parameters, the second image region separate from the first image region and to be displayed in the second screen region, the second set of encoding parameters resulting in a lower quality than the first set of encoding parameters, and
responsive to a difference between a depth of the object in the image and the user's depth of focus exceeding a threshold value, encoding the object, including the portion to be displayed within the first screen region, with the second set of encoding parameters; and

transmitting the encoded image to the display device to be displayed on the screen.

2. The method of claim 1 , wherein the first screen region corresponds to a region of the screen viewed by a fovea of the eye.

3. The method of claim 1 , wherein the second screen region corresponds to a region of the screen viewed by a portion of the retina of the eye outside of the fovea.

4. The method of claim 1 , wherein the first set of encoding parameters includes a first image resolution, the second set of encoding parameters includes a second image resolution, and the second image resolution is lower than the first image resolution.

5. The method of claim 4 , wherein encoding the first image region comprises generating a first encoded image region, wherein encoding the second image region comprises generating a second encoded image region, and further comprising:
generating a packed image comprising the first encoded image region and the second encoded image region.

6. The method of claim 1 , wherein the first set of encoding parameters includes a first framerate, the second set of encoding parameters includes a second framerate, and the second framerate is lower than the first framerate.

7. The method of claim 1 , further comprising:
determining, based on the eye tracking information, a third screen region corresponding to a region of the screen that is separate from the first screen region and the second screen region,
wherein encoding the image for display on the screen further comprises:
encoding a third image region of the image based on a third set of encoding parameters, the third image region separate from the first image region and the second image region and to be displayed in the third screen region, the third set of encoding parameters resulting in a lower quality than the first set of encoding parameters and a higher quality than the second set of encoding parameters.

8. The method of claim 7 , wherein the third screen region corresponds to a region of the screen viewed by a parafovea of the eye.

9. The method of claim 1 , wherein the display device is a virtual reality headset.

10. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving eye tracking information comprising at least one image of an eye of a user viewing a screen of a display device;
determining, based on the eye tracking information, a first screen region corresponding to a region of the screen, the first screen region containing a point of regard of the user, the point of regard representing a point on the screen at which the user is looking at the time the eye tracking information was captured;
determining, based on the eye tracking information, a second screen region corresponding to a region of the screen that is separate from the first screen region;
determining, based on the eye tracking information, a user's depth of focus,
encoding an image for display on the screen, the image comprising an object with at least a portion of the object to be displayed within the first screen region, the encoding comprising:
encoding a first image region of the image based on a first set of encoding parameters, the first image region comprising a portion of the image to be displayed in the first screen region,
encoding a second image region of the image based on a second set of encoding parameters, the second image region separate from the first image region and to be displayed in the second screen region, the second set of encoding parameters resulting in a lower quality than the first set of encoding parameters, and
responsive to a difference between a depth of the object in the image and the user's depth of focus exceeding a threshold value, encoding the object, including the portion to be displayed within the first screen region, with the second set of encoding parameters; and

transmitting the encoded image to the display device to be displayed on the screen.

11. The computer-readable storage medium of claim 10 , wherein the first screen region corresponds to a region of the screen viewed by a fovea of the eye.

12. The computer-readable storage medium of claim 10 , wherein the second screen region corresponds to a region of the screen viewed by a portion of the retina of the eye outside of the fovea.

13. The computer-readable storage medium of claim 10 , wherein the first set of encoding parameters includes a first image resolution, the second set of encoding parameters includes a second image resolution, and the second image resolution is lower than the first image resolution.

14. The computer-readable storage medium of claim 10 , wherein encoding the first image region comprises generating a first encoded image region, wherein encoding the second image region comprises generating a second encoded image region, and further comprising:
generating a packed image comprising the first encoded image region and the second encoded image region.

15. The computer-readable storage medium of claim 10 , wherein the first set of encoding parameters includes a first framerate, the second set of encoding parameters includes a second framerate, and the second framerate is lower than the first framerate.

16. The computer-readable storage medium of claim 10 , the operations further comprising:
determining, based on the eye tracking information, a third screen region corresponding to a region of the screen that is separate from the first screen region and the second screen region,
wherein encoding the image for display on the screen further comprises:
encoding a third image region of the image based on a third set of encoding parameters, the third image region separate from the first image region and the second image region and to be displayed in the third screen region, the third set of encoding parameters resulting in a lower quality t

Assignees

• Facebook Tech Llc

Inventors

• Tall Martin Henrik
• San Agustin Lopez Javier
• Dahl Rasmus

Classifications

• H04N19/124

  Quantisation · CPC title

• H04N19/162

  User input · CPC title

• H04N19/17

  the unit being an image region, e.g. an object · CPC title

• H04N19/167

  Position within a video image, e.g. region of interest [ROI] · CPC title

• G09G5/14

  Display of multiple viewports · CPC title