Displaying Representations of Environments

Patent metadata

Field	Value
Publication number	US-2023400956-A1
Application number	US-202318238297-A
Country	US
Kind code	A1
Filing date	Aug 25, 2023
Priority date	Sep 25, 2019
Publication date	Dec 14, 2023
Grant date	—

Abstract

A method includes displaying a home ER environment characterized by home ER world coordinates, including one or more diorama-view representation of one or more respective ER environments. Each diorama-view representation includes ER objects arranged in a spatial relationship according to corresponding ER world coordinates. In some implementations, in response to detecting an input directed to a first diorama-view representation, the method includes transforming the home ER environment. Transforming the home ER environment includes transforming the spatial relationship between a subset of the ER objects as a function of the home ER world coordinates and corresponding ER world coordinates. In some implementations, in response to detecting an input associated with a first one of a plurality of diorama-view representations, the method includes changing display of the first one of the plurality of diorama-view representations from a first viewing vector to a second viewing vector while maintaining an arrangement of ER objects.

First claim

What is claimed is:

1 . A method comprising:
at an electronic device including one or more processors, a non-transitory memory, one or more input devices, and a display device:
displaying, via the display device, a home enhanced reality (ER) environment characterized by home ER world coordinates, including a first diorama-view representation of a first ER environment, wherein the first diorama-view representation includes one or more of ER objects arranged in a spatial relationship according to first ER world coordinates;
detecting, via the one or more input devices, a first input that is directed to the first diorama-view representation; and
in response to detecting the first input, transforming the home ER environment by:
ceasing to display the first diorama-view representation within the home ER environment,
transforming the spatial relationship between a subset of the one or more ER objects as a function of the home ER world coordinates and the first ER world coordinates, and
displaying, via the display device, the subset of the one or more ER objects within the home ER environment based on the transformation.

2 . The method of claim 1 , further comprising:
while displaying the subset of the one or more ER objects within the home ER environment based on the transformation, detecting, via the one or more input devices, a second input; and in response to detecting the second input, adding the subset of the one or more ER objects to the home ER environment.

3 . The method of claim 1 , further comprising:
while displaying the subset of the one or more ER objects within the home ER environment based on the transformation, detecting, via the one or more input devices, a second input; and in response to detecting the second input, replacing the home ER environment with the first ER environment that includes the subset of the one or more ER objects.

4 . The method of claim 1 , wherein, in response to detecting the first input, displaying the subset of the one or more ER objects from a first viewing vector, the method further comprising:
while displaying the subset of the one or more ER objects within the home ER environment based on the transformation, detecting, via the one or more input devices, a second input; and in response to detecting the second input, changing display of the subset of the one or more ER objects from the first viewing vector to a second viewing vector while maintaining the subset of the one or more ER objects arranged according to the first ER world coordinates.

5 . The method of claim 4 , wherein the second input is directed to the subset of the one or more ER objects.

6 . The method of claim 5 , wherein the one or more input devices includes a hand tracking sensor, the method further comprising:
detecting the second input via the hand tracking sensor; obtaining hand tracking data from the hand tracking sensor based on the second input; and determining, from the hand tracking data, that the second input is directed to the subset of the one or more ER objects.

7 . The method of claim 6 , wherein the one or more input devices includes an eye tracking sensor, the method further comprising:
detecting the second input via the eye tracking sensor; obtaining eye tracking data from the eye tracking sensor based on the second input; and determining, from the eye tracking data, that the second input is directed to the subset of the one or more ER objects.

8 . The method of claim 5 , wherein the home ER environment includes one or more physical objects that are associated with the home ER world coordinates, and wherein changing display of the subset of the one or more ER objects from the first viewing vector to the second viewing vector includes moving the subset of the one or more ER objects relative to the one or more physical objects.

9 . The method of claim 5 , wherein the second input corresponds to a change in position of the electronic device from a first pose to a second pose relative to the subset of the one or more ER objects, and wherein changing display of the subset of the one or more ER objects from the first viewing vector to the second viewing vector is based on the change in position of the electronic device from the first pose to the second pose.

10 . The method of claim 1 , further comprising:
obtaining, via an image sensor, environmental data bounded by a field-of-view associated with the image sensor, wherein the environmental data is associated with a physical environment including one or more physical objects; identifying, within the environmental data, a particular one of the one or more physical objects located within a spatial proximity threshold of the subset of the one or more ER objects; and moving the subset of the one or more ER objects relative to the one or more physical objects based on the particular one of the one or more physical objects.

11 . The method of claim 1 , further comprising:
displaying, via the display device, a plurality of diorama-view representations of a corresponding plurality of ER environments within the home ER environment, wherein the plurality of diorama-view representations includes the first diorama-view representation; and in response to detecting the first input, selecting the first diorama-view representation from the plurality of diorama-view representations.

12 . The method of claim 1 , further comprising:
displaying, via the display device, a plurality of diorama-view representations of a corresponding plurality of ER environments within the home ER environment, wherein the plurality of diorama-view representations includes the first diorama-view representation; and detecting, via the one or more input devices, a selection input that selects the first diorama-view representation from the plurality of diorama-view representations; and in response to detecting the selection input, maintaining display of the first diorama-view representation within the home ER environment and ceasing to display the remainder of the plurality of diorama-view representations.

13 . The method of claim 12 , further comprising obtaining, via an eye tracking sensor, eye gaze data indicative of an eye gaze location, wherein the selection input is based on the eye gaze location.

14 . The method of claim 1 , wherein the first ER environment is associated with an ER session that enables respective graphical representations of individuals to be concurrently within the first ER environment.

15 . The method of claim 14 , wherein the first ER environment includes one or more ER representations respectively associated with one or more individuals that are connected to the ER session.

16 . The method of claim 15 , wherein each of the one or more individuals has a respective access level that satisfies an access level criterion that is associated with the ER session.

17 . The method of claim 15 , further comprising:
while displaying the first diorama-view representation:
playing, via a speaker of the electronic device, a first set of speech data that is associated with the one or more individuals that are connected to the ER session;
obtaining, via an audio sensor of the electronic device, a second set of speech data from a user that is associated with the electronic device; and
providing the second set of speech data to the one or more individuals that are connected to the ER session.

18 . The method of claim 1 , further comprising:
obtaining a characterization vector that provides a spatial characterization of the first ER environmen

Assignees

• Apple Inc

Inventors

• Sommer Bruno M
• Morgan Tucker Bull
• Chiu Shih Sang
• Smith Connor Alexander

Classifications

• G06F3/04815 Primary

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/16

  Sound input; Sound output (speech processing G10L) · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title