Methods and systems for displaying additional content on a heads up display displaying a virtual reality environment

Patent metadata

Field	Value
Publication number	US-9851792-B2
Application number	US-201615140249-A
Country	US
Kind code	B2
Filing date	Apr 27, 2016
Priority date	Apr 27, 2016
Publication date	Dec 26, 2017
Grant date	Dec 26, 2017

Abstract

Methods and systems for presenting additional content in virtual reality environments on heads up displays showing main content without interfering with a user's viewing of the main content. Specifically, a movement of a center of gaze of the user is detected. A visual field of the user based on the movement of the center of gaze is determined. If it is determined that the portion of the virtual reality environment in which the additional content is generated for display corresponds to a foreground area of the user's visual field, the additional content is generated for display in a portion of the virtual reality environment corresponding to a peripheral area of the user's visual field.

First claim

I claim:

1. A method for presenting additional content in virtual reality environments on heads up displays showing main content without interfering with a user's viewing of the main content comprising:
generating for display, in a first portion of a virtual reality environment in a heads up display, first main content, wherein the first portion corresponds to a foreground area of a first visual field of a user;
generating for display, in a second portion of the virtual reality environment in the heads up display, second main content and additional content, wherein the second portion corresponds to a peripheral area of the first visual field of the user;
detecting a movement of a center of gaze of the user;
determining a second visual field of the user based on the movement of the center of gaze;
determining that the second portion corresponds to a foreground area of the second visual field; and
in response to determining that the second portion corresponds to the foreground area of the second visual field, generating for display the additional content in a third portion of the virtual reality environment, wherein the third portion corresponds to a peripheral area of the second visual field.

2. The method of claim 1 , wherein detecting the movement of the center of gaze comprises detecting the center of gaze of the user.

3. The method of claim 2 , wherein detecting the center of gaze of the user comprises:
transmitting light to each eye of the user;
collecting an image of each eye of the user;
detecting, in each image, a location of a reflection in an eye of the user;
determining a location of each pupil of the user;
comparing the location of each pupil to the location of each reflection;
based on comparing the location of each pupil to the location of each reflection, determining a gaze point of each eye of the user; and
determining the center of gaze by determining a midpoint between the gaze point of each eye of the user.

4. The method of claim 1 , wherein the heads up display is a head-mounted display physically coupled to the head of the user, wherein the head-mounted display includes an accelerometer, and wherein detecting the movement of the center of gaze of the user comprises detecting, by the accelerometer, an acceleration of the head of the user.

5. The method of claim 1 , wherein the heads up display is a head-mounted display physically coupled to the head of the user, wherein the head-mounted display includes a camera feeding a video of the user's physical surroundings to the head-mounted display, and wherein generating for display the additional content comprises generating for display the video of the user's physical surroundings.

6. The method of claim 1 , wherein generating for display the additional content comprises generating for display the additional content as a picture-in-picture.

7. The method of claim 1 , wherein determining the second visual field of the user based on the movement of the center of gaze comprises:
determining a new center of gaze based on the movement of the center of gaze; and
determining an area that is within a first number of degrees to the right and left of the new center of gaze, a second number of degrees above the new center of gaze, and a third number of degrees below the new center of gaze.

8. The method of claim 7 , wherein determining that the second portion corresponds to the foreground area of the second visual field comprises determining the foreground area of the second visual field, and wherein determining the foreground area of the second visual field comprises:
determining an area that is within a fourth number of degrees to the right and left of the new center of gaze, a fifth number of degrees above the new center of gaze, and a sixth number of degrees below the new center of gaze, wherein the fourth number is smaller than the first number, the fifth number is smaller than the second number, and the sixth number is smaller than the third number.

9. The method of claim 1 , further comprising:
receiving an input from the user; and
in response to the input, generating for display the additional content in a portion of the virtual reality environment corresponding to a foreground area of the second visual field.

10. The method of claim 1 , wherein generating for display the additional content comprises:
measuring a time period after the movement of the center of gaze of the user during which the center of gaze of the user has not substantially moved;
determining that the time period is greater than a threshold time period; and
in response to determining that the time period is greater than the threshold time period, generating for display the additional content.

11. A system for presenting additional content in virtual reality environments on heads up displays showing main content without interfering with a user's viewing of the main content comprising:
control circuitry configured to:
generate for display, in a first portion of a virtual reality environment in a heads up display, first main content, wherein the first portion corresponds to a foreground area of a first visual field of a user;
generate for display, in a second portion of the virtual reality environment in the heads up display, second main content and additional content, wherein the second portion corresponds to a peripheral area of the first visual field of the user;
determine a second visual field of the user based on a movement of a center of gaze of the user;
determine that the second portion corresponds to a foreground area of the second visual field; and
in response to determining that the second portion corresponds to the foreground area of the second visual field, generate for display the additional content in a third portion of the virtual reality environment, wherein the third portion corresponds to a peripheral area of the second visual field; and

a detection module configured to detect the movement of the center of gaze of the user.

12. The system of claim 11 , wherein the detection module is further configured to detect the movement of the center of gaze by detecting the center of gaze of the user.

13. The system of claim 12 , wherein the detection module is further configured, when detecting the center of gaze of the user, to:
transmit light to each eye of the user;
collect an image of each eye of the user;
detect, in each image, a location of a reflection in an eye of the user;
determine a location of each pupil of the user;
compare the location of each pupil to the location of each reflection;
based on comparing the location of each pupil to the location of each reflection, determine a gaze point of each eye of the user; and
determine the center of gaze by determining a midpoint between the gaze point of each eye of the user.

14. The system of claim 11 , wherein the heads up display is a head-mounted display physically coupled to the head of the user, wherein the head-mounted display includes an accelerometer, and wherein the detection module is further configured, when detecting the movement of the center of gaze of the user, to detect, by the accelerometer, an acceleration of the head of the user.

15. The system of claim 11 , wherein the heads up display is a head-mounted display physically coupled to the head of the user, wherein the head-mounted display includes a camera feeding a video of the user's physical surroundings to the head-mounted display, and wherein the control circuitry is further configured, when generating for display the additional content, to generate for display the video of the user's physical surroundings.

Assignees

• Rovi Guides Inc

Inventors

• Shanware Ajit

Classifications

• G02B27/0093

  with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06T2207/30201

  Face · CPC title

• G02B27/017

  Head mounted · CPC title