Scene understanding using occupancy grids

Patent metadata

Field	Value
Publication number	US-12444136-B2
Application number	US-202218275468-A
Country	US
Kind code	B2
Filing date	Feb 3, 2022
Priority date	Feb 4, 2021
Publication date	Oct 14, 2025
Grant date	Oct 14, 2025

Abstract

This document describes scene understanding for cross reality systems using occupancy grids. In one aspect, a method includes recognizing one or more objects in a model of a physical environment generated using images of the physical environment. For each object, a bounding box is fit around the object. An occupancy grid that includes a multiple cells is generated within the bounding box around the object. A value is assigned to each cell of the occupancy grid based on whether the cell includes a portion of the object. An object representation that includes information describing the occupancy grid for the object is generated. The object representations are sent to one or more devices.

First claim

What is claimed is:

1. A computer-implemented method performed by one or more data processing apparatus, the method, comprising:
recognizing one or more objects in a model of a physical environment generated using images of the physical environment;
for each object of the one or more objects:
fitting a bounding box around each object;
generating an occupancy grid within the bounding box around each object, wherein the occupancy grid includes a plurality of cells;
assigning a value to each cell of the occupancy grid based on whether the cell includes a portion of each object; and
generating an object representation that includes information describing the occupancy grid for each object; and

sending the object representations to one or more devices.

2. The computer-implemented method of claim 1 , wherein assigning a value to each cell of the occupancy grid based on whether the cell includes a portion of each object, comprises:
assigning a first value to each cell that includes a portion of each object; and
assigning a second value different from the first value to each cell that does not include any portion of each object.

3. The computer-implemented method of claim 1 , comprising:
detecting a change to a given object of the one or more objects;
generating a new occupancy grid for the given object; and
sending the new occupancy grid for the given object to the one or more devices rather than an updated model of the physical environment.

4. The computer-implemented method of claim 3 , wherein each device of the one or more devices update a local mesh for the physical environment using the new occupancy grid for the given object.

5. The computer-implemented method of claim 1 , wherein the model comprises a plurality of voxels that represent the physical environment.

6. The computer-implemented method of claim 5 , further comprising assigning a semantic label to each voxel based on a type of object recognized in the voxel.

7. The computer-implemented method of claim 6 , further comprising clustering voxels based on the semantic label for each voxel.

8. A computer-implemented system, comprising:
one or more computers; and
one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, perform operations comprising:
recognizing one or more objects in a model of a physical environment generated using images of the physical environment;
for each object of the one or more objects:
fitting a bounding box around each object;
generating an occupancy grid within the bounding box around each object, wherein the occupancy grid includes a plurality of cells;
assigning a value to each cell of the occupancy grid based on whether the cell includes a portion of each object; and
generating an object representation that includes information describing the occupancy grid for each object; and

sending each object representations to one or more devices.

9. The computer-implemented system of claim 8 , wherein assigning a value to each cell of the occupancy grid based on whether the cell includes a portion of each object, comprises:
assigning a first value to each cell that includes a portion of each object; and
assigning a second value different from the first value to each cell that does not include any portion of each object.

10. The computer-implemented system of claim 8 , wherein the operations comprise:
detecting a change to a given object of the one or more objects;
generating a new occupancy grid for the given object; and
sending the new occupancy grid for the given object to the one or more devices rather than an updated model of the physical environment.

11. The computer-implemented system of claim 10 , wherein each device of the one or more devices update a local mesh for the physical environment using the new occupancy grid for the given object.

12. The computer-implemented system of claim 8 , wherein the model comprises a plurality of voxels that represent the physical environment.

13. The computer-implemented system of claim 12 , wherein the operations comprise assigning a semantic label to each voxel based on a type of object recognized in the voxel.

14. A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform operations, comprising:
recognizing one or more objects in a model of a physical environment generated using images of the physical environment;
for each object of the one or more objects:
fitting a bounding box around each object;
generating an occupancy grid within the bounding box around each object, wherein the occupancy grid includes a plurality of cells;
assigning a value to each cell of the occupancy grid based on whether the cell includes a portion of each object; and
generating an object representation that includes information describing the occupancy grid for each object; and

sending each object representations to one or more devices.

15. The non-transitory, computer-readable medium of claim 14 , wherein assigning a value to each cell of the occupancy grid based on whether the cell includes a portion of each object, comprises:
assigning a first value to each cell that includes a portion of each object; and
assigning a second value different from the first value to each cell that does not include any portion of each object.

16. The non-transitory, computer-readable medium of claim 14 , wherein the operations comprise:
detecting a change to a given object of the one or more objects;
generating a new occupancy grid for the given object; and
sending the new occupancy grid for the given object to the one or more devices rather than an updated model of the physical environment.

17. The non-transitory, computer-readable medium of claim 16 , wherein each device of the one or more devices update a local mesh for the physical environment using the new occupancy grid for the given object.

18. The non-transitory, computer-readable medium of claim 14 , wherein the model comprises a plurality of voxels that represent the physical environment.

19. The non-transitory, computer-readable medium of claim 18 , wherein the operations comprise assigning a semantic label to each voxel based on a type of object recognized in the voxel.

20. The non-transitory, computer-readable medium of claim 19 , wherein the operations comprise clustering voxels based on the semantic label for each voxel.

Assignees

• Magic Leap Inc

Inventors

• Ramnath Divya
• Dong Shiyu
• Choudhary Siddharth
• Mahendran Siddharth
• Kannan Arumugam Kalai
• SINGHAL Prateek
• Gupta Khushi

Classifications

• G06T2210/12

  Bounding box · CPC title

• G06V20/64

  Three-dimensional [3D] objects · CPC title

• G06V20/70

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title