Scene graph-based scene re-identification

Patent metadata

Field	Value
Publication number	US-11640708-B1
Application number	US-202117227724-A
Country	US
Kind code	B1
Filing date	Apr 12, 2021
Priority date	Apr 16, 2020
Publication date	May 2, 2023
Grant date	May 2, 2023

Abstract

Implementations disclosed herein provide systems and methods that match a current scene graph associated with a user's current environment to a prior scene graph for a prior environment to determine that the user is in the same environment. The current scene graph is compared with the prior scene graph based on matching the objects within the current scene graph with objects within the prior scene graph.

First claim

What is claimed is:

1. A method comprising:
at a processor:
generating a scene graph for a current environment based on one or more images of the current environment, the scene graph for the current environment representing objects within the current environment and relationships between the objects within the current environment
comparing the scene graph for the current environment with a scene graph for a prior environment based on matching the objects within the scene graph for the current environment with objects within the scene graph for the prior environment;
determining that the current environment matches the prior environment based on the matching; and
based on said determining that the current environment matches the prior environment, determining that a current location associated with the user is within the prior environment.

2. The method of claim 1 further comprising comparing object metadata associated with the objects within the scene graph for the current environment with object metadata associated with the objects within the scene graph for the prior environment.

3. The method of claim 1 further comprising comparing object-specific meshes associated with the objects within the scene graph for the current environment with object-specific meshes associated with the objects within the scene graph for the prior environment.

4. The method of claim 1 further comprising comparing object-specific keyframes associated with the objects within the scene graph for the current environment with object-specific keyframes associated with the objects within the scene graph for the prior environment.

5. The method of claim 1 further comprising comparing object location information or relationships associated with the objects within the scene graph for the current environment with object location information or relationships associated with the objects within the scene graph for the prior environment.

6. The method of claim 1 , wherein one or more types of objects are excluded from the comparison of the scene graph for the current environment with the scene graph for the prior environment.

7. The method of claim 6 , wherein object types that are transient are excluded from the comparison of the scene graph for the current environment with the scene graph for the prior environment.

8. The method of claim 1 , wherein matching the objects within the scene graph for the current environment with the objects within the scene graph for the prior environment includes a multi-step matching process.

9. The method of claim 8 , wherein the multi-step matching process includes refining a matching score.

10. The method of claim 1 , wherein the multi-step matching process is discontinued based on criteria or a low score.

11. The method of claim 1 further comprising identifying that a user is in the same room based on the determination that the current environment matches the prior environment.

12. The method of claim 1 further comprising identifying that a user is in a similar room based on the determination that the current environment matches the prior environment.

13. The method of claim 1 further comprising enhancing the current environment based on the prior environment and the determination that the current environment matches the prior environment.

14. A system comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
generating a scene graph for a current environment based on one or more images of the current environment, the scene graph for the current environment representing objects within the current environment and relationships between the objects within the current environment;
comparing the scene graph for the current environment with a scene graph for a prior environment based on matching the objects within the scene graph for the current environment with objects within the scene graph for the prior environment;
determining that the current environment matches the prior environment based on the matching; and
based on said determining that the current environment matches the prior environment, determining that a current location associated with the user is within the prior environment.

15. The system of claim 14 , the operations further comprising comparing object metadata associated with the objects within the scene graph for the current environment with object metadata associated with the objects within the scene graph for the prior environment.

16. The system of claim 14 , the operations further comprising comparing object-specific meshes associated with the objects within the scene graph for the current environment with object-specific meshes associated with the objects within the scene graph for the prior environment.

17. The system of claim 14 , the operations further comprising comparing object-specific keyframes associated with the objects within the scene graph for the current environment with object-specific keyframes associated with the objects within the scene graph for the prior environment.

18. The system of claim 14 , the operations further comprising comparing object location information or relationships associated with the objects within the scene graph for the current environment with object location information or relationships associated with the objects within the scene graph for the prior environment.

19. The system of claim 14 , wherein one or more types of objects are excluded from the comparison of the scene graph for the current environment with the scene graph for the prior environment.

20. A non-transitory computer-readable storage medium, storing program instructions computer-executable on a computer to perform operations comprising:
generating a scene graph for a current environment based on one or more images of the current environment, the scene graph for the current environment representing objects within the current environment and relationships between the objects within the current environment; and
comparing the scene graph for the current environment with a scene graph for a prior environment based on matching the objects within the scene graph for the current environment with objects within the scene graph for the prior environment;
determining that the current environment matches the prior environment based on the matching; and
based on said determining that the current environment matches the prior environment, determining that a current location associated with the user is within the prior environment.

Assignees

• Apple Inc

Inventors

• Blechschmidt Angela
• Polichroniadis Alexander S
• Ulbricht Daniel

Classifications

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06V20/46

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title

• G06V20/10 Primary

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06F18/29

  Graphical models, e.g. Bayesian networks · CPC title

• G06K9/6215

  Physics · mapped topic