Devices, methods, systems, and media for an extended screen distributed user interface in augmented reality

Patent metadata

Field	Value
Publication number	US-12093704-B2
Application number	US-202117228087-A
Country	US
Kind code	B2
Filing date	Apr 12, 2021
Priority date	Apr 12, 2021
Publication date	Sep 17, 2024
Grant date	Sep 17, 2024

Abstract

Devices, methods, systems, and media are described for providing an extended screen distributed user interface in an augmented reality environment. GUI layout information for laying out a conventional 2D GUI is processed in order to generate an extended screen DUI for display partially on a 2D display device and partially on one or more virtual screens of an AR environment viewed using an AR display device, such as a head mounted display. GUI elements are laid out in the DUI based on a primary modality of the GUI element (input or output), and/or based on spatial dependencies between GUI elements encoded in the GUI layout information. Methods for switching focus between two software application instances displayed in the DUI are also disclosed.

First claim

The invention claimed is:

1. A method for displaying a distributed user interface (DUI), comprising:
obtaining graphical user interface (GUI) information of a software application comprising:
GUI element information for each of a plurality of GUI elements of the software application; and
GUI layout information for arranging the plurality of GUI elements for presentation as a GUI screen on a two-dimensional (2D) display, the GUI layout information comprising a predetermined semantic structure that defines relationships among the plurality of GUI elements, the predetermined semantic structure being used to define a spatial layout of the plurality of GUI elements when presented on the 2D display, the predetermined semantic structure being defined by the software application;

processing the GUI layout information to identify, based at least in part on the semantic structure, a first subset of the plurality of GUI elements for display on a primary DUI screen and a second subset of the plurality of GUI elements for display on a secondary DUI screen;
displaying a primary DUI screen on a 2D display device, the primary DUI screen including the GUI element information of the first subset of the plurality of GUI elements;
obtaining 2D display device location information indicating a physical location of the 2D display device;
determining DUI layout information indicating a fixed spatial relationship between the physical location of the 2D display device for displaying the first subset of the plurality of GUI elements and a secondary DUI screen virtual location for displaying the second subset of the plurality of GUI elements, the DUI layout information being determined by:
generating a directed acyclic graph (DAG) based on the predetermined semantic structure of the GUI layout information, wherein:
each vertex of the DAG corresponding to a GUI module having one or more GUI elements of the plurality of GUI elements; and
each edge of the DAG represents a spatial dependency between two GUI modules corresponding to two vertices connected by the edge;

topologically sorting the vertices of the DAG based on the spatial dependencies of the DAG to generate a sorted order of the vertices; and
generating the fixed spatial relationship based on the sorted order of the vertices;

wherein the fixed spatial relationship of the DUI layout information preserves the predetermined semantic structure defining relationships between the first subset and the second subset of the plurality of GUI elements;
determining the secondary DUI screen virtual location based on the physical location of the 2D display device and the fixed spatial relationship; and
displaying a view of an augmented reality (AR) environment on an AR display device, the AR environment including a secondary DUI screen located at the determined secondary DUI screen virtual location.

2. The method of claim 1 , wherein the fixed spatial relationship comprises:
an angular position of the secondary DUI screen such that the secondary DUI screen is displayed substantially co-planar with the 2D display device; and
a linear position of the secondary DUI screen such that the secondary DUI screen is displayed proximate to, and non-overlapping with, the 2D display device.

3. The method of claim 1 , wherein displaying the view of the AR environment on the AR display device comprises:
obtaining 2D display device display surface area information indicating a display surface area of the 2D display device;
processing the 2D display device display surface area information to generate secondary DUI screen virtual size information; and
displaying the view of an AR environment on the AR display device such that the secondary DUI screen is displayed with a virtual size indicated by the secondary DUI screen virtual size information.

4. The method of claim 1 , further comprising:
processing the GUI layout information to identify one or more additional subsets of the plurality of GUI elements for display on one or more respective additional DUI screens;
generating one or more additional DUI screens, each additional DUI screen comprising the GUI element information for each GUI element of one of the one or more additional subsets; and
displaying the one or more additional DUI screens on the AR display device.

5. The method of claim 4 , wherein displaying the one or more additional DUI screens on the AR display device comprises:
determining additional DUI layout information indicating:
for each additional DUI screen, an additional fixed spatial relationship between the physical location of the 2D display device and a respective additional DUI screen virtual location for displaying each respective additional subset of the plurality of GUI elements, wherein the additional fixed spatial relationship of the additional DUI layout information preserves the predetermined semantic structure defining relationships between the first subset of the plurality of GUI elements and each respective additional subset of the plurality of GUI elements;

for each additional DUI screen, determining the respective additional DUI screen virtual location based on the physical location of the 2D display device and the respective additional fixed spatial relationship; and
displaying a view of the AR environment on the AR display device, the AR environment further including the one or more additional DUI screens, each additional DUI screen being located at its respective additional DUI screen virtual location.

6. The method of claim 5 , wherein:
the primary DUI screen and the one or more additional DUI screens comprise GUI information for a first software application instance;
the secondary DUI screen comprises GUI information for a second software application instance; and
the method further comprises:
processing user input information to detect user attention directed to the second software application instance; and
swapping the GUI information for the first software application instance on the primary DUI screen and the one or more additional DUI screens with the GUI information for the second software application on the secondary DUI screen.

7. The method of claim 1 , wherein the GUI layout information comprises one or more layout definition files, each layout file comprising one or more semantic tags.

8. The method of claim 1 , wherein processing the GUI layout information to identify the first subset and second subset comprises:
processing the GUI layout information to identify a primary modality of each of a plurality of GUI modules, each GUI module comprising a subset of the plurality of GUI elements;
identifying the second subset of the plurality of GUI elements as a GUI module having a primary modality of output; and
identifying the first subset of the plurality of GUI elements as a GUI module having a primary modality of input.

9. The method of claim 1 , wherein:
the GUI information comprises:
GUI information for a first software application instance; and
GUI information for a second software application instance;

the primary DUI screen comprises a first DUI screen of the first software application instance;
the secondary DUI screen comprises a first DUI screen of the second software application instance;
the method further comprising:
processing user input information to detect user attention directed to the secondary DUI screen;
displaying the secondary DUI screen on the 2D display device; and
displaying a view of the AR environment on the AR display device, the AR environment including the primary DUI screen located at the secondary DUI screen virtual location.

10. The method of claim 1 , wherein:
the GUI information comprises:
GUI information for a firs

Assignees

• Khan Taslim Arefin
• Fan Szu Wen
• Zou Changqing
• Xu Jianpeng
• Li Wei
• Huawei Tech Co Ltd

Inventors

• Khan Taslim Arefin
• Fan Szu Wen
• Zou Changqing
• Xu Jianpeng
• Li Wei

Classifications

• G06F16/90

  Details of database functions independent of the retrieved data types · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/1423

  controlling a plurality of local displays, e.g. CRT and flat panel display · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title