Depth-assigned content for depth-enhanced virtual reality images

Patent metadata

Field	Value
Publication number	US-10129524-B2
Application number	US-201715439710-A
Country	US
Kind code	B2
Filing date	Feb 22, 2017
Priority date	Jun 26, 2012
Publication date	Nov 13, 2018
Grant date	Nov 13, 2018

Abstract

According to various embodiments of the invention, a system and method are provided for enabling interaction with, manipulation of, and control of depth-assigned content in depth-enhanced pictures, such as virtual reality images. Depth-assigned content can be assigned to a specified depth value. When a depth-enhanced picture is refocused at a focus depth substantially different from the specified assigned depth value, the depth-assigned content may be omitted, grayed out, blurred, or otherwise visually distinguished. In this manner, content associated with an in-focus image element can be visually distinguished from content associated with an out-of-focus image element. For example, in at least one embodiment, depth-assigned content is visible only when an image element associated with the content is in focus (or nearly in focus). According to various embodiments of the invention, many different types of interactions are facilitated among depth-assigned content, depth-enhanced virtual reality images, and other content.

First claim

What is claimed is:

1. A method for manipulating depth-assigned content associated with a depth-enhanced virtual reality image, comprising:
at a processor, receiving a depth-enhanced virtual reality image comprising a plurality of elements having different depth values;
at the processor, receiving a depth-assigned content item, the depth-assigned content item being associated with the depth-enhanced virtual reality image and being assigned to a first depth value;
at the processor, determining a focus depth for rendering of the depth-enhanced virtual reality image, wherein at least a first one of the elements has a depth that is closer to the focus depth than the depth of at least a second one of the elements;
at the processor, determining whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image based on a determination as to whether the first depth value falls within a predetermined tolerance value of the determined focus depth;
at an image rendering component, rendering the depth-enhanced virtual reality image so that the first element is more in focus than the second element;
at the image rendering component:
responsive to the determination indicating that the depth-assigned content item should be included, including the depth-assigned content item in the rendered depth-enhanced virtual reality image;
responsive to the determination indicating that the depth-assigned content item should not be included, omitting the depth-assigned content item from the rendered depth-enhanced virtual reality image; and

outputting the rendered depth-enhanced virtual reality image at an output device.

2. The method of claim 1 , wherein the depth-enhanced virtual reality image comprises a computer-generated image.

3. The method of claim 1 , wherein the depth-enhanced virtual reality image comprises a combination of a captured image and a computer-generated image.

4. The method of claim 1 , wherein:
the depth-assigned content item is assigned to a location within the depth-enhanced virtual reality image; and
rendering the depth-enhanced virtual reality image to generate a rendered image including the depth-assigned content item comprises rendering the depth-assigned content item at the location to which it is assigned.

5. The method of claim 4 , wherein the position within the depth-enhanced virtual reality image is indicated as a coordinate value in three-dimensional space.

6. The method of claim 1 , wherein determining a focus depth comprises receiving, at an input device, user input specifying a focus depth.

7. The method of claim 1 , wherein determining a focus depth comprises automatically determining the focus depth.

8. The method of claim 1 , wherein determining whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image comprises determining whether the first depth value equals the determined focus depth.

9. The method of claim 1 , wherein rendering the depth-enhanced virtual reality image to generate a rendered image including the depth-assigned content item comprises:
responsive to the first depth value falling outside the predetermined tolerance value of the determined focus depth, rendering the depth-enhanced virtual reality image to generate a rendered image including a subdued representation of the depth-assigned content item.

10. The method of claim 9 , wherein the subdued representation of the depth-assigned content item comprises at least one selected from the group consisting of:
a grayed out representation of the depth-assigned content item; and
a blurred representation of the depth-assigned content item.

11. The method of claim 1 , further comprising:
at the processor, determining a second focus depth for rendering of the depth-enhanced virtual reality image;
at the processor, determining whether to include the depth-assigned content item in rendering the depth-enhanced virtual reality image, based on a determination as to whether the first depth value falls within a predetermined tolerance value of the determined second focus depth;
at the image rendering component, responsive to the determination indicating that the depth-assigned content item should be included, and responsive to the previous determination indicating that the depth-assigned content item should not be included, re-rendering the depth-enhanced virtual reality image to introduce the depth-assigned content item into the rendered image;
at the image rendering component, responsive to the determination indicating that the depth-assigned content item should not be included, and responsive to the previous determination indicating that the depth-assigned content item should be included, re-rendering the depth-enhanced virtual reality image to dismiss the depth-assigned content item from the rendered image;
outputting the rendered image at an output device.

12. The method of claim 1 , wherein the depth-assigned content item comprises at least one selected from the group consisting of:
a tag;
an annotation;
a comment;
a text item;
an image;
an interactive element;
a hyperlink;
a reference to a document; and
a reference to a resource.

13. The method of claim 1 , wherein the depth-assigned content item comprises a machine-readable code.

14. The method of claim 1 , wherein the depth-assigned content item comprises at least one selected from the group consisting of:
in-image content; and
off-image content.

15. The method of claim 1 , further comprising:
at a user input device, receiving user input indicating an interaction with the depth-enhanced virtual reality image; and
responsive to the interaction with the depth-enhanced virtual reality image, causing a change to the depth-assigned content item.

16. The method of claim 15 , wherein causing a change to the depth-assigned content item comprises causing at least one additional content item to be displayed.

17. The method of claim 15 , wherein receiving user input indicating an interaction with the depth-enhanced virtual reality image comprises collecting information about user interaction with the depth-enhanced virtual reality image, the method further comprising:
determining at least one point of interest based on the collected information.

18. The method of claim 15 , wherein receiving user input indicating an interaction with the depth-enhanced virtual reality image comprises determined a direction of user gaze, the method further comprising:
determining at least one point of interest based on the determined direction of user gaze.

19. The method of claim 1 , further comprising:
at a user input device, receiving user input indicating an interaction with the depth-assigned content item; and
responsive to the interaction with the depth-enhanced virtual reality image, causing a change to the depth-enhanced virtual reality image.

20. The method of claim 19 , wherein causing a change to the depth-enhanced virtual reality image comprises re-rendering the depth-enhanced virtual reality image at a depth responsive to the received user input.

21. The method of claim 20 , wherein re-rendering the depth-enhanced virtual reality image at a depth responsive to the received user input comprises bringing into focus an area indicated to be of interest based on the received user input.

22. The method of claim 20 , wherein re-rendering the depth-enhanced virtual reality image at a depth responsive to the received user input comprises

Assignees

• Google Llc

Inventors

• Ng Yi-Ren
• Cheng Eric
• Liang Chia-Kai
• Fatahalian Kayvon
• Evans David John
• Wampler Kira
• Berman Kristen
• Akeley Kurt Barton

Classifications

• G06F21/31

  User authentication · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• H04N13/128 Primary

  Adjusting depth or disparity · CPC title

• G06T15/205 Primary

  Image-based rendering · CPC title

• G06T2200/21

  involving computational photography · CPC title