Selective rendering of sparse peripheral displays based on element saliency

Patent metadata

Field	Value
Publication number	US-10216260-B2
Application number	US-201715470773-A
Country	US
Kind code	B2
Filing date	Mar 27, 2017
Priority date	Mar 27, 2017
Publication date	Feb 26, 2019
Grant date	Feb 26, 2019

Abstract

Peripheral visualizations are based on various attributes associated with a scene. Characteristics of elements in a scene are determined. Based on these characteristics, the salience of the elements is determined. When the element is salient, then this determination also includes a saliency magnitude of the element. Thereafter, the embodiments determine whether the element's saliency magnitude exceeds a particular saliency threshold. If the magnitude does exceed this threshold, then the embodiments render a corresponding peripheral visualization with the peripheral display(s) proximate the salient element(s).

First claim

What is claimed is:

1. A computer system used for display of computer-generated mixed-reality, virtual-reality or augmented-reality scenes comprising:
one or more peripheral display portions surrounding a primary display;
one or more processors; and
one or more computer-readable hardware storage media having stored thereon computer-executable instructions, the computer-executable instructions being executable by the one or more processors to cause the computer system to selectively control when a peripheral display portion is activated to render a peripheral visualization based on various attributes associated with a given computer-generated scene by causing the computer system to:
set all pixels in the one or more peripheral display portions to a normally off configuration;
render the given computer-generated scene;
determine characteristics of at least one element associated with the given computer-generated scene, wherein the characteristics describe i) a role of the at least one element based on an event that involves the at least one element in relation to the given computer-generated scene, or ii) relative position that indicates whether the at least one element is either near or intersecting with an edge region of at least one peripheral display portion;
set a saliency threshold based on at least one or more of the determined characteristics, wherein the saliency threshold is either dynamically adjustable based on content included in the given computer-generated scene, or is adjustable by a user of the computer system;
based on one or more of the determined characteristics used to determine the saliency threshold, determine that the at least one element is salient, and determine a saliency magnitude of the at least one element;
determine that the at least one element's saliency magnitude meets or exceeds the saliency threshold; and
after determining that the at least one element's saliency magnitude meets or exceeds the saliency threshold, activate the pixels of the at least one peripheral display portion so as to render the peripheral visualization of the at least one element on the at least one peripheral display portion.

2. The computer system of claim 1 , wherein the role of the at least one element in relation to the given computer-generated scene includes an appearance event.

3. The computer system of claim 1 , wherein the role of the at least one element in relation to the given computer-generated scene includes a motion event.

4. The computer system of claim 1 , wherein the role of the at least one element in relation to the given computer-generated scene includes a disappearance event.

5. The computer system of claim 1 , wherein the primary display and the one or more peripheral display portions together comprise a wrap-around foveated single display, and wherein the given computer-generated scene is displayed on the wrap-around foveated single display.

6. The computer system of claim 1 , wherein an extent to which the peripheral visualization is rendered is based on the determined characteristics of the at least one element.

7. The computer system of claim 6 , wherein the extent is further based on the determined saliency magnitude of the at least one element.

8. The computer system of claim 1 , wherein the determined characteristics of the at least one element comprise size of the at least one element, and wherein the size of the at least one element is translated to an angular space for rendering.

9. The computer system of claim 8 , wherein the peripheral visualization is rendered abased on one or more of 1) the translated angular space of the at least one element, 2) a color of the at least one element, and 3) one or more parameters of an event that is associated with the at least one element.

10. One or more hardware storage devices having stored thereon computer-executable instructions that are executable by one or more processors of a computer system used for display of computer-generated mixed-reality, virtual-reality or augmented-reality scenes and wherein the computer system comprises one or more peripheral display portions surrounding a primary display, and wherein the computer-executable instructions cause the computer system to selectively control when a peripheral display portion is activated to render a peripheral visualization based on various attributes associated with a given computer-generated scene by causing the computer system to:
set all pixels in the one or more peripheral display portions to a normally off configuration;
render the given computer-generated scene;
determine characteristics of at least one element associated with the given computer-generated scene, wherein the characteristics describe i) a role of the at least one element based on an event that involves the at least one element in relation to the given computer-generated scene, and ii) relative position that indicates whether the at least one element is either near or intersecting with an edge region of at least one peripheral display portion;
set a saliency threshold based on at least one or more of the determined characteristics, wherein the saliency threshold is either dynamically adjustable based on content included in the given computer-generated scene, or is adjustable by a user of the computer system;
based on one or more of the determined characteristics used to determine the saliency threshold, determine that the at least one element is salient, and determine a saliency magnitude of the at least one element;
determine that the at least one element's saliency magnitude meets or exceeds the saliency threshold; and
after determining that the at least one element's saliency magnitude meets or exceeds the saliency threshold, activate the pixels of the at least one peripheral display portion so as to render the peripheral visualization of the at least one element on the at least one peripheral display portion.

11. The one or more hardware storage devices of claim 10 , wherein the one or more peripheral display portions comprise a pixel density less that the primary display's pixel density.

12. The one or more hardware storage devices of claim 11 , wherein the one or more peripheral display portions comprise one or more light emitting diodes (LEDs), wherein the determined characteristics of the at least one element comprise a color of the at least one element, and wherein rendering the peripheral visualization comprises causing at least one LED of the one or more LEDs to light up according to the color of the element.

13. The one or more hardware storage devices of claim 10 , wherein the determined characteristics used to determine saliency further comprise on one or more of 1) a historical trend of a user using the computer system, 2) a present configuration of the computer system, 3) a determined reaction time of the user, and 4) an eye movement of the user.

14. The one or more hardware storage devices of claim 10 , wherein the primary display and the one or more peripheral display portions together comprise a wrap-around foveated single display, and wherein the given computer-generated scene is displayed on the wrap-around foveated single display.

15. The one or more hardware storage devices of claim 10 , wherein the determined characteristics of the at least one element comprise one or more of 1) a size of the at least one element, 2) a brightness of the at least one element, 3) a flashing status of the at least one element, and 4) a color content of the at least one element.

16. The one or more hardware storage devices of claim 10 , wherein the determined characteristics of the

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Crisler Matthew Calbraith
• Held Robert Thomas
• Glik Eliezer

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• A63F2300/8082

  Virtual reality · CPC title

• G06T13/40

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06K9/00456

  Physics · mapped topic