Generating a semantic construction of a physical setting

Patent metadata

Field	Value
Publication number	US-11893207-B2
Application number	US-202117475004-A
Country	US
Kind code	B2
Filing date	Sep 14, 2021
Priority date	Apr 23, 2019
Publication date	Feb 6, 2024
Grant date	Feb 6, 2024

Abstract

In some implementations, a method includes obtaining environmental data corresponding to a physical environment. In some implementations, the method includes determining, based on the environmental data, a bounding surface of the physical environment. In some implementations, the method includes detecting a physical element located within the physical environment based on the environmental data. In some implementations, the method includes determining a semantic label for the physical element based on at least a portion of the environmental data corresponding to the physical element. In some implementations, the method includes generating a semantic construction of the physical environment based on the environmental data. In some implementations, the semantic construction of the physical environment includes a representation of the bounding surface, a representation of the physical element and the semantic label for the physical element.

First claim

What is claimed is:

1. A method comprising:
at a device including a non-transitory memory and one or more processors coupled with the non-transitory memory:
obtaining environmental data corresponding to a physical environment;
determining, based on the environmental data, a bounding surface of the physical environment;
detecting a physical element located within the physical environment based on the environmental data;
determining a semantic label for the physical element based on at least a portion of the environmental data corresponding to the physical element;
generating a semantic construction of the physical environment based on the environmental data, wherein the semantic construction of the physical environment includes a representation of the bounding surface, a representation of the physical element and the semantic label for the physical element; and
providing the semantic construction of the physical environment to a virtual intelligent agent (VIA) that generates actions for a graphical object that represents the VIA.

2. The method of claim 1 , wherein detecting the physical element comprises performing instance segmentation on the environmental data in order to detect the physical element.

3. The method of claim 1 , wherein detecting the physical element comprises identifying an optical machine-readable representation of data associated with the physical element.

4. The method of claim 3 , wherein the optical machine-readable representation of data includes a barcode, and wherein obtaining the environmental data comprises scanning the barcode.

5. The method of claim 1 , wherein determining the semantic label comprises performing semantic segmentation on at least a portion of the environmental data corresponding to the physical element in order to determine the semantic label for the physical element.

6. The method of claim 1 , wherein determining the semantic label comprises identifying one or more properties associated with the physical element, and selecting the semantic label based on the one or more properties associated with the physical element.

7. The method of claim 1 , wherein determining the semantic label comprises performing an image search based on a portion of the environmental data corresponding to the physical element, and receiving the semantic label as a search result.

8. The method of claim 1 , wherein generating the semantic construction comprises determining a placement of the representation of the physical element in relation to the representation of the bounding surface.

9. The method of claim 8 , wherein determining the placement of the representation of the physical element comprises determining an orientation of the representation of the physical element within the semantic construction.

10. The method of claim 1 , further comprising:
generating, based on the semantic construction of the physical environment, a graphical setting that corresponds to the physical environment, wherein the graphical setting includes a graphical object that represents the physical element.

11. The method of claim 10 , further comprising:
displaying the graphical setting with the graphical object.

12. The method of claim 1 , further comprising:
providing the semantic construction of the physical environment to an objective-effectuator engine that generates actions for a graphical object representing an objective-effectuator that is instantiated in a graphical setting.

13. The method of claim 1 , wherein the bounding surface includes a physical surface in the physical environment.

14. The method of claim 13 , wherein the bounding surface includes a floor, a ceiling, or a wall in the physical environment.

15. The method of claim 1 , wherein determining the bounding surface includes identifying a boundary associated with the physical environment and representing the boundary with a representation of a surface in the semantic construction of the physical environment.

16. The method of claim 15 , wherein identifying the boundary includes identifying a plot line associated with the physical environment based on information stored in a datastore.

17. The method of claim 15 , further comprising:
adding a representation of a wall in the semantic construction along the boundary.

18. A device comprising:
one or more processors;
a non-transitory memory;
one or more displays; and
one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to:
obtain environmental data corresponding to a physical environment;
determine, based on the environmental data, a bounding surface of the physical environment;
detect a physical element located within the physical environment based on the environmental data;
determine a semantic label for the physical element based on at least a portion of the environmental data corresponding to the physical element;
generate a semantic construction of the physical environment based on the environmental data, wherein the semantic construction of the physical environment includes a representation of the bounding surface, a representation of the physical element and the semantic label for the physical element; and
provide the semantic construction of the physical environment to a virtual intelligent agent (VIA) that generates actions for a graphical object that represents the VIA.

19. The device of claim 18 , wherein determining the semantic label comprises performing semantic segmentation on at least a portion of the environmental data corresponding to the physical element in order to determine the semantic label for the physical element.

20. The device of claim 18 , wherein determining the semantic label comprises identifying one or more properties associated with the physical element, and selecting the semantic label based on the one or more properties associated with the physical element.

21. The device of claim 18 , wherein determining the semantic label comprises performing an image search based on a portion of the environmental data corresponding to the physical element, and receiving the semantic label as a search result.

22. A non-transitory memory storing one or more programs, which, when executed by one or more processors of a device with a display, cause the device to:
obtain environmental data corresponding to a physical environment;
determine, based on the environmental data, a bounding surface of the physical environment;
detect a physical element located within the physical environment based on the environmental data;
determine a semantic label for the physical element based on at least a portion of the environmental data corresponding to the physical element;
generate a semantic construction of the physical environment based on the environmental data, wherein the semantic construction of the physical environment includes a representation of the bounding surface, a representation of the physical element and the semantic label for the physical element; and
provide the semantic construction of the physical environment to an objective-effectuator engine that generates actions fora graphical object representing an objective-effectuator that is instantiated in a graphical setting.

23. The non-transitory memory of claim 22 , wherein the one or more programs which, when executed by the one or more processors of the device with the display, further cause the device to:
provide the semantic construction of the physical environm

Assignees

• Apple Inc

Inventors

• Drummond Mark
• MORGAN Bo
• Sivapurapu Siva Chandra Mouli

Classifications

• G06F3/04815 Primary

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06F16/53

  Querying · CPC title

• G06K7/1413

  1D bar codes · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06V10/225

  based on a marking or identifier characterising the area · CPC title