Location-based virtual element modality in three-dimensional content

Patent metadata

Field	Value
Publication number	US-12211152-B2
Application number	US-202217727311-A
Country	US
Kind code	B2
Filing date	Apr 22, 2022
Priority date	Sep 11, 2018
Publication date	Jan 28, 2025
Grant date	Jan 28, 2025

Abstract

Various implementations disclosed herein include devices, systems, and methods that enable a device to provide a view of virtual elements and a physical environment where the presentation of the virtual elements is based on positioning relative to the physical environment. In one example, a device is configured to detect a change in positioning of a virtual element, for example, when a virtual element is added, moved, or the physical environment around the virtual element is changed. The location of the virtual element in the physical environment is used to detect an attribute of the physical environment upon which the presentation of the virtual element depends. Thus, the device is further configured to detect an attribute (e.g., surface, table, mid-air, etc.) of the physical environment based on the placement of the virtual element and present the virtual element based on the detected attribute.

First claim

What is claimed is:

1. A method comprising:
at a device with a processor and a computer readable medium:
obtaining a request to display an application at a first location in a three-dimensional (3D) environment, wherein the application comprises a modality state definition for each of a plurality of positional states, and wherein each modality state definition comprises a different two-dimensional (2D) or 3D model representation of the application, wherein each different 2D or 3D model representation of the application comprises a different functionality and a different interactive feature;
determining a first positional state associated with the first location in the 3D environment;
displaying, at the first location in the 3D environment, a first 2D or 3D model representation of the application that corresponds to the modality state definition matching the determined first positional state, wherein the displaying of the first 2D or 3D model representation of the application comprises a visual representation of a first associated functionality and a visual representation of a first associated interactive feature;
obtaining a second request to display the application at a second location different than the first location;
determining a second positional state associated with the second location in the 3D environment;
determining, based on the second positional state, whether to update the first 2D or 3D model representation of the application; and
in accordance with determining to update the first 2D or 3D model representation of the application, displaying, at the second location in the 3D environment, a second 2D or 3D model representation of the application that corresponds to the modality state definition matching the determined second positional state, wherein the displaying of the second 2D or 3D model representation of the application comprises a visual representation of a second associated functionality that is different than the first associated functionality and a visual representation of a second associated interactive feature that is different than the first associated interactive feature.

2. The method of claim 1 , wherein determining the first positional state comprises:
identifying a classification of a portion of the 3D environment using a classifier; and
identifying an amount of the portion available upon which the first 2D or 3D model representation of the application can be positioned.

3. The method of claim 1 , wherein, in accordance with determining the first positional state, the method further comprises updating the first 2D or 3D model representation of the application to have a first visual appearance when updating the first 2D or 3D model representation of the application; and
wherein in accordance with determining the second positional state, the method further comprises updating the second 2D or 3D model representation of the application to have a second visual appearance when updating the second 2D or 3D model representation of the application, wherein the first visual appearance is different from the second visual appearance.

4. The method of claim 1 , wherein, in accordance with determining the first positional state, the method further comprises updating the first 2D or 3D model representation of the application to be updated to have a first functionality when updating the first 2D or 3D model representation of the application; and
wherein in accordance with determining the second positional state, the method further comprises updating the second 2D or 3D model representation of the application to be updated to have a second functionality when updating the second 2D or 3D model representation of the application, wherein the first functionality is different from the second functionality.

5. The method of claim 1 , wherein, in accordance with determining the first positional state, the method further comprises displaying an updated 2D or 3D model representation of the application at a second location; and
wherein in accordance with determining the second positional state, the method further comprises displaying the updated 2D or 3D model representation of the application away from the second location, wherein the first positional state is different from the second positional state.

6. The method of claim 1 , wherein the modality state definition of the first 2D or 3D model representation of the application is based on a function of the first 2D or 3D model representation of the application.

7. The method of claim 1 , wherein displaying the first 2D or 3D model representation of the application comprises:
obtaining, using an image sensor, an image of the 3D environment;
generating a representation of the 3D environment using the obtained image; and
presenting the first 2D or 3D model representation of the application within 3D environment.

8. The method of claim 1 , wherein displaying the first 2D or 3D model representation of the application comprises:
identifying, using an image sensor, a feature in the 3D environment; and
presenting the first 2D or 3D model representation of the application on an optical see-through display responsive to identifying the feature.

9. The method of claim 1 , further comprising:
receiving user input representing a change in the first location of the first 2D or 3D model representation of the application or the second 2D or 3D model representation of the application.

10. The method of claim 1 , wherein determining the first positional state associated with the first location in the 3D environment or determining the second positional state associated with the second location in the 3D environment comprises determining that the first 2D or 3D model representation of the application is on a planar surface.

11. The method of claim 10 , wherein determining the first positional state associated with the first location in the 3D environment or determining the second positional state associated with the second location in the 3D environment comprises determining that the planar surface is a wall, table, floor, ramp, stairway, or ceiling.

12. The method of claim 1 , wherein determining the first positional state associated with the first location in the 3D environment or determining the second positional state associated with the second location in the 3D environment comprises determining that the first 2D or 3D model representation of the application or the second 2D or 3D model representation of the application is on a horizontal planar surface.

13. The method of claim 1 , wherein the first positional state associated with the first location in the 3D environment or determining the second positional state associated with the second location in the 3D environment determining that the first 2D or 3D model representation of the application or the second 2D or 3D model representation of the application is on a vertical planar surface.

14. A system comprising:
an electronic device with a display and an image sensor;
a processor; and
a computer-readable storage medium comprising instructions that upon execution by the processor cause the processor to perform operations, the operations comprising:
obtaining a request to display an application at a first location in a three-dimensional (3D) environment, wherein the application comprises a modality state definition for each of a plurality of positional states, and wherein each modality state definition comprises a different two-dimensional (2D) or 3D model representation of the application, wherein each different 2D or 3D model representation of the application comprises a different functionality and a different interactiv

Assignees

• Apple Inc

Inventors

• Burns Aaron M
• Sommer Bruno M
• Oriol Timothy R

Classifications

• G02B27/0172

  characterised by optical features · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G06T2219/2004

  Aligning objects, relative positioning of parts · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title