Dynamically modifying virtual and augmented reality content to reduce depth conflict between user interface elements and video content

Patent metadata

Field	Value
Publication number	US-10957063-B2
Application number	US-201815935976-A
Country	US
Kind code	B2
Filing date	Mar 26, 2018
Priority date	Mar 26, 2018
Publication date	Mar 23, 2021
Grant date	Mar 23, 2021

Abstract

The present disclosure relates to systems, methods, and non-transitory computer readable media for generating modified video content to reduce depth conflicts between user interface elements and video objects. For example, the disclosed systems can analyze an input video to identify feature points that designate objects within the input video and to determine the depths of the identified feature points. In addition, the disclosed systems can compare the depths of the feature points with a depth of a user interface element to determine whether there are any depth conflicts. In response to detecting a depth conflict, the disclosed systems can modify the depth of the user interface element to reduce or avoid the depth conflict. Furthermore, the disclosed systems can apply a blurring effect to an area around a user interface element to reduce the effect of depth conflicts.

First claim

What is claimed is:

1. In a digital medium environment for rendering virtual reality content, a non-transitory computer readable medium for reducing depth conflicts while providing user interface elements within video content comprising instructions that, when executed by a processor, cause a computer device to:
analyze an input video to identify feature points within a region of the input video that is less than a viewable area of the input video, the region defined as an area within a threshold angle between a vector that originates at a user position and intersects a user interface element and another vector that originates at the user position and that, when rotated, intersects points outlining the region of the input video around the user interface element;
determine, based on analyzing the input video, depths of the feature points within the region of the input video;
compare the depths of the feature points with a depth of the user interface element within the input video; and
dynamically modify, based on the comparison of the depths of the feature points with the depth of the user interface element, the depth of the user interface element to avoid depth conflicts with the feature points.

2. The non-transitory computer readable medium of claim 1 , wherein the instructions cause the computer device to determine the depths of the feature points by determining, for each feature point, a screen disparity between a left view of the feature point within the input video and a right view of the feature point within the input video.

3. The non-transitory computer readable medium of claim 1 , wherein the instructions further cause the computer device to determine the depth of the user interface element by determining a screen disparity between a left view of a center point of the user interface element and a right view of the center point of the user interface element.

4. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the processor, cause the computer device to:
determine a depth of the region of the input video defined as the area within the threshold angle averaging the depths of the feature points within the region; and
wherein the instructions cause the computer device to compare the depths of the feature points with the depth of the user interface element by determining a difference between the depth of the region and the depth of the user interface element.

5. The non-transitory computer readable medium of claim 4 , wherein the instructions cause the computer device to modify the depth of the user interface element within the input video by minimizing a disparity between the depth of the user interface element and the depth of the region of the input video.

6. The non-transitory computer readable medium of claim 4 , wherein the instructions cause the computer device to modify the depth of the user interface element by adjusting a frustum associated with a left view of the user interface element within the input video and a frustum associated with a right view of the user interface element within the input video.

7. The non-transitory computer readable medium of claim 6 , further comprising instructions that, when executed by the processor, cause the computer device to detect that the user interface element is moving slower than a movement threshold; and
wherein the instructions cause the computer device to, in response to detecting that the user interface element is moving slower than the movement threshold, adjust the frustum associated with the left view and the frustum associated with the right view at a default rate.

8. The non-transitory computer readable medium of claim 6 , further comprising instructions that, when executed by the processor, cause the computer device to detect that the user interface element is moving faster than a movement threshold; and
wherein the instructions cause the computer device to, in response to detecting that the user interface element is moving faster than the movement threshold, adjust the frustum associated with the left view and the frustum associated with the right view at a rate faster than a default rate.

9. In a digital medium environment for rendering virtual reality content, a system for reducing depth conflicts while providing user interface elements within video content comprising:
a processor; and
a non-transitory computer readable medium comprising instructions that, when executed by the processor, cause the system to:
analyze an input video to identify feature points within a region of the input video that is less than a viewable area of the input video, the region defined as an area within a threshold angle between a vector that originates at a user position and intersects a user interface element and another vector that originates at the user position and that, when rotated, intersects points outlining the region of the input video around the user interface element;
determine, based on analyzing the input video, depths of the feature points within the region of the input video by calculating screen disparities between a left view of the feature points and a right view of the feature points;
compare the depths of the feature points with a depth of the user interface element within the input video by determining a difference between the depths of the feature points and the depth of the user interface element; and
dynamically modify, based on the comparison of the depths of the feature points with the depth of the user interface element, the depth of the user interface element to avoid depth conflicts with the feature points.

10. The system of claim 9 , wherein the instructions further cause the system to determine the depth of the user interface element by determining a screen disparity between a left view of a center point of the user interface element and a right view of the center point of the user interface element.

11. The system of claim 9 , further comprising instructions that, when executed by the processor, cause the system to:
determine a depth of the region of the input video defined as the area within the threshold angle based on the depths of the feature points within the region.

12. The system of claim 11 , wherein the instructions cause the system to compare the depths of the feature points with the depth of the user interface element by determining a difference between the depth of the region of the input video and the depth of the user interface element.

13. The system of claim 9 , further comprising instructions that, when executed by the processor, cause the system to detect that there is a depth conflict between the user interface element and the region of the input video within the threshold angle; and
wherein the instructions cause the system to modify, based on detecting the depth conflict, the depth of the user interface element within the input video by decreasing the depth of the user interface element.

14. The system of claim 9 , further comprising instructions that, when executed by the processor, cause the system to determine that there is no depth conflict between the user interface element and the region of the input video within the threshold angle; and
wherein the instructions cause the system to modify, based on determining that there is no depth conflict, the depth of the user interface element within the input video by increasing the depth of the user interface element.

15. The system of claim 14 , further comprising instructions that, when executed by the processor, cause the system to detect that the user interface element is moving slower than a mov

Assignees

• Adobe Inc
• Univ Portland State

Inventors

• Diverdi Stephen
• Nguyen Cuong
• Hertzmann Aaron
• Liu Feng

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title

• H04N13/183

  On-screen display [OSD] information, e.g. subtitles or menus · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title