Modeling interactions between simulated characters and real-world objects for more realistic augmented reality

Patent metadata

Field	Value
Publication number	US-10445940-B2
Application number	US-201815922820-A
Country	US
Kind code	B2
Filing date	Mar 15, 2018
Priority date	Mar 15, 2018
Publication date	Oct 15, 2019
Grant date	Oct 15, 2019

Abstract

A simulation engine models interactions between a simulated character and real-world objects to produce a physically realistic augmented reality (AR) simulation. The simulation engine recognizes a given real-world object and then identifies, within a library of object models, an object model corresponding to that object. The simulation engine projects the object model onto the real-world object such that the object model is geometrically aligned with the real-world object. When the simulated character encounters the real-world object, the simulation engine models interactions between the simulated character and the real-world object by adjusting the kinematics of the simulated character relative to the object model associated with the real-world object.

First claim

What is claimed is:

1. A computer-implemented method for generating simulated characters within augmented reality (AR) simulations, the method comprising:
mapping a first real-world object to a first object model, wherein the first object model includes first geometry corresponding to the structure of the first real-world object;
projecting the first geometry onto the first real-world object within an AR simulation;
determining that a first portion of a simulated character included in the AR simulation occupies a first position within the AR simulation that is associated with the first real-world object; and
generating a kinematic model of the simulated character based on the first object model, wherein the kinematic model of the simulated character models an interaction between the simulated character and the first real-world object, and wherein the simulated character is subsequently displayed to a user via an AR headset.

2. The computer-implemented method of claim 1 , further comprising generating the first object model by:
capturing video imagery of the first real-world object from a plurality of perspectives; and
analyzing the video imagery to determine a set of physical dimensions associated with the first real-world object.

3. The computer-implemented method of claim 2 , further comprising analyzing the video imagery to generate a first texture map that describes a first surface of the first object model.

4. The computer-implemented method of claim 1 , wherein mapping the first real-world object to the first object model comprises:
scanning a first region of space to identify the first real-world object;
determining a first set of features associated with the first real-world object; and
determining that the first object model includes at least one feature included in the first set of features.

5. The computer-implemented method of claim 1 , wherein determining that the first portion of the simulated character occupies a first position within the AR simulation comprises determining that the first portion of the simulated character intersects the first object model.

6. The computer-implemented method of claim 1 , wherein generating the kinematic model of the simulated character comprises performing an inverse kinematic operation on a previous kinematic model of the simulated character based on the first geometry.

7. The computer-implemented method of claim 1 , wherein the first real-world object comprises an inclined plane, and generating the kinematic model of the simulated character causes the simulated character to ascend the first object model within the AR simulation.

8. The computer-implemented method of claim 1 , wherein the first real-world object comprises a dynamically moving object, and generating the kinematic model of the simulated character causes the simulated character to move in response to a collision with the first object model.

9. The computer-implemented method of claim 1 , wherein the first real-world object comprises a static object, and generating the kinematic model of the simulated character comprises applying a ragdoll model of the simulated character to a previous kinematic model of the simulated character in response to a collision between the simulated character and the first object model.

10. The computer-implemented method of claim 1 , wherein generating the kinematic model of the simulated character causes the simulated character to extract a simulated element from the first object model.

11. A non-transitory computer-readable medium storing program instructions that, when executed by a processor, cause the processor to generate simulated characters within augmented reality (AR) simulations by performing the steps of:
mapping a first real-world object to a first object model, wherein the first object model includes first geometry corresponding to the structure of the first real-world object;
projecting the first geometry onto the first real-world object within an AR simulation;
determining that a first portion of a simulated character included in the AR simulation occupies a first position within the AR simulation that is associated with the first real-world object; and
generating a kinematic model of the simulated character based on the first object model, wherein the kinematic model of the simulated character models an interaction between the simulated character and the first real-world object, and wherein the simulated character is subsequently displayed to a user via an AR headset.

12. The non-transitory computer-readable medium of claim 11 , further comprising generating the first object model by:
capturing video imagery of the first real-world object from a plurality of perspectives; and
analyzing the video imagery to determine a set of physical dimensions associated with the first real-world object.

13. The non-transitory computer-readable medium of claim 12 , further comprising the step of analyzing the video imagery to generate a first texture map that describes a first surface of the first object model.

14. The non-transitory computer-readable medium of claim 11 , wherein the step of mapping the first real-world object to the first object model comprises:
scanning a first region of space to identify the first real-world object;
determining a first set of features associated with the first real-world object; and
determining that the first object model includes at least one feature included in the first set of features.

15. The non-transitory computer-readable medium of claim 11 , wherein the step of determining that the first portion of the simulated character occupies a first position within the AR simulation comprises determining that the first portion of the simulated character intersects the first object model.

16. The non-transitory computer-readable medium of claim 11 , wherein the step of generating the kinematic model of the simulated character comprises performing an inverse kinematic operation on a previous kinematic model of the simulated character based on the first geometry.

17. The non-transitory computer-readable medium of claim 11 , wherein the step of generating the kinematic model of the simulated character causes the simulated character to extract a simulated element from the first object model.

18. The non-transitory computer-readable medium of claim 11 , wherein the step of generating the kinematic model of the simulated character prevents any portion of the simulated character from occupying a second position within the AR simulation that is associated with the first real-world object.

19. The non-transitory computer-readable medium of claim 11 , wherein the AR simulation is associated with a first region of space that includes the first real-world object.

20. A system, comprising:
a memory storing a simulation engine; and
a processor that, when executing the simulation engine, is configured to perform the steps of:
mapping a first real-world object to a first object model, wherein the first object model includes first geometry corresponding to the structure of the first real-world object,
projecting the first geometry onto the first real-world object within an AR simulation,
determining that a first portion of a simulated character included in the AR simulation occupies a first position within the AR simulation that is associated with the first real-world object, and
generating a kinematic model of the simulated character based on the first object model, wherein the kinematic model of the simulated character models an intera

Assignees

• Disney Entpr Inc
• Eth Zuerich

Inventors

• Guay Martin
• Çimen Gökçen
• Borer Dominik Tobias
• Guggiari Simone
• Yuan Ye
• Coros Stelian
• Sumner Robert Walker

Classifications

• G06T13/40

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• G06T2215/16

  Using real world measurements to influence rendering · CPC title