Switching mode of operation in a head mounted display

Patent metadata

Field	Value
Publication number	US-10019057-B2
Application number	US-201414206219-A
Country	US
Kind code	B2
Filing date	Mar 12, 2014
Priority date	Jun 7, 2013
Publication date	Jul 10, 2018
Grant date	Jul 10, 2018

Abstract

Methods, systems, and computer programs are presented for managing the display of images on a head mounted device (HMD). One method includes an operation for tracking the gaze of a user wearing the HMD, where the HMD is displaying a scene of a virtual world. In addition, the method includes an operation for detecting that the gaze of the user is fixed on a predetermined area for a predetermined amount of time. In response to the detecting, the method fades out a region of the display in the HMD, while maintaining the scene of the virtual world in an area of the display outside the region. Additionally, the method includes an operation for fading in a view of the real world in the region as if the HMD were transparent to the user while the user is looking through the region. The fading in of the view of the real world includes maintaining the scene of the virtual world outside the region.

First claim

What is claimed is:

1. A method for managing display of images on a head mounted device (HMD), the method comprising:
presenting video data on a display of the HMD, the video data providing a view of a virtual world;
tracking a gaze of a user wearing the HMD that is displaying the video data;
detecting that the gaze of the user is fixed on a predetermined area of the display for a predetermined amount of time, which triggers modification to the video data presented at the display, without requiring user action, wherein the modification of the video data includes,
fading out a portion of the video data that is being rendered in the predetermined area of the display of the HMD while maintaining the view of the virtual world in an area of the display outside the predetermined area; and
fading in a portion of an image of a real world in the predetermined area, wherein the image of the real world is captured by a forward facing camera of the HMD and the portion of the image of the real world corresponds to a direction of the gaze of the user, the fading in simulating a see-through mode of the display in the predetermined area by extending a view of the user through the HMD into the real world in the direction of the gaze, while the area outside of the predetermined area continues to present the view of the virtual world,

wherein the fading out of the video data and the fading in of the image of the real world is performed for a period of time.

2. The method as recited in claim 1 , wherein the predetermined area, where the gaze of the user has been detected for a predetermined amount of time, is one of an area defined below the display, an area defined to a side of the display, an area defined above the display, or an area defined on a corner of the display.

3. The method as recited in claim 1 , wherein the predetermined area, where the gaze of the user has been detected for a predetermined amount of time, is defined below the display, and wherein the region is a bottom half of the display.

4. The method as recited in claim 1 , wherein the predetermined area is a rectangle centered on the display.

5. The method as recited in claim 1 , further including:
providing an option for user confirmation before fading in the portion of the image of the real world in the predetermined area.

6. The method as recited in claim 1 , further including:
receiving a command from the user to terminate rendering of the image of the real world;
in response to the command, fading out the portion of the image of the real world in the predetermined area of the display; and
fading in the portion of the virtual world that corresponds with the predetermined area while keeping the view of the virtual world in the area outside the predetermined area.

7. The method as recited in claim 1 , wherein tracking the gaze further includes:
capturing images of eyes of the user with a camera inside the HMD; and
analyzing the captured images to determine the gaze of the user.

8. The method as recited in claim 1 , wherein the predetermined amount of time is two seconds.

9. The method of claim 1 , further comprising,
detecting a notification from a device of the user, wherein the view in the direction of the gaze is toward the device.

10. The method of claim 9 , wherein the notification is a phone call or a message being received via the device.

11. A method for managing images rendered on a display of a head mounted device (HMD), the method comprising:
detecting an operation to acclimate a user to the HMD;
presenting a view of a real world on the display to simulate a see-through mode for the display of the HMD, the view of the real world captured by a camera disposed on a front side of the HMD;
adding a virtual element from a virtual world to the view of the real world to simulate an augmented reality mode for the display, wherein the virtual element is added in an area on the display by fading out a portion of the view of the real world rendered in the area and fading in the virtual element, such that the view of the real world is not visible in the area occupied by the virtual element;
after adding the virtual element in the area, fading out a portion of the view of the real world that corresponds with a second area of the display that is outside the area occupied by the virtual element while maintaining the virtual element on the display in the area without fading the virtual element; and
fading in a portion of a view of the virtual world that corresponds with the second area of the display while keeping the virtual element, wherein fading in the portion of the view of the virtual world includes adding additional virtual elements that form the virtual world, wherein the virtual element is present during the fading out the portion of the view of the real world and the fading in of the virtual world to assist the user to acclimate into the view of the virtual world.

12. The method as recited in claim 11 , wherein the augmented reality mode includes presenting the virtual element as if the virtual element was located in the real world, wherein a motion of the HMD with respect to the real world causes a change in the view of the real world while maintaining the virtual element in a same location in the real world.

13. The method as recited in claim 11 , wherein the real world and the virtual world are not blended on the display except for the virtual element.

14. The method as recited in claim 11 , wherein the real world and the virtual world are blended during the fade out of the real world.

15. The method as recited in claim 11 , wherein operations of the method are performed by a computer program when executed by one or more processors, the computer program being embedded in a non-transitory computer-readable storage medium.

16. A non-transitory computer-readable storage medium storing a computer program, the computer-readable storage medium comprising:
program instructions for presenting video data on a display of the HMD, the video data providing a view of a virtual world;
program instructions for tracking a gaze of a user wearing the HMD that is displaying the video data;
program instructions for detecting that the gaze of the user is fixed on a predetermined area of the display for a predetermined amount of time, which triggers modification to the video data presented at the display without requiring user action, wherein the modification of the video data includes,
program instructions for fading out a portion of the video data that is being rendered in the predetermined area of the display of the HMD while maintaining the view of the virtual world in an area of the display outside the predetermined area; and
program instructions for fading in a portion of an image of a real world in the predetermined area, wherein the image of the real world is captured by a forward facing camera of the HMD and the portion of the image of the real world corresponds to a direction of the gaze of the user, the fading in simulating a see-through mode of the display in the predetermined area by extending a view of the user through the HMD into the real world in the direction of the gaze, while the area outside of the predetermined area continues to present the view of the virtual world,

wherein the fading out of the video data and the fading in of the image of the real world is performed for a period of time.

17. The computer program as recited in claim 16 , wherein the predetermined area is one of an area defined below the display, an area defined to a side of the display, an area defined above the

Assignees

• Sony Interactive Entertainment Inc

Inventors

• Osman Steven
• Mao Xiadong
• Stafford Jeffrey Roger

Classifications

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G02B27/0093

  with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking · CPC title

• G06F2203/04804

  Transparency, e.g. transparent or translucent windows · CPC title