Matching meshes for virtual avatars

Patent metadata

Field	Value
Publication number	US-11074748-B2
Application number	US-201916274677-A
Country	US
Kind code	B2
Filing date	Feb 13, 2019
Priority date	Feb 27, 2018
Publication date	Jul 27, 2021
Grant date	Jul 27, 2021

Abstract

Examples of systems and methods for matching a base mesh to a target mesh for a virtual avatar or object are disclosed. The systems and methods may be configured to automatically match a base mesh of an animation rig to a target mesh, which may represent a particular pose of the virtual avatar or object. Base meshes may be obtained by manipulating an avatar or object into a particular pose, while target meshes may be obtain by scanning, photographing, or otherwise obtaining information about a person or object in the particular pose. The systems and methods may automatically match a base mesh to a target mesh using rigid transformations in regions of higher error and non-rigid deformations in regions of lower error.

First claim

What is claimed is:

1. A system for matching a base mesh for a virtual avatar to a target mesh of the virtual avatar, the system comprising:
non-transitory computer storage configured to store a target mesh for the virtual avatar and a base mesh for the virtual avatar, the target mesh comprising a plurality of target mesh vertices and the base mesh comprising a plurality of base mesh vertices; and
a hardware processor in communication with the non-transitory computer storage, the hardware processor programmed to:
(i) determine a plurality of error regions between the target mesh and the base mesh, each error region representing a deviation in distance between positions of the base mesh vertices relative to positions of the target mesh vertices, the plurality of error regions comprising a first error region having a first error and a second error region having a second error, the second error being less than the first error;
(ii) apply a rigid transformation to the base mesh vertices in the first error region, wherein applying the rigid transformation does not change a size and shape of the first error region;
(iii) apply a non-rigid transformation to the base mesh vertices in the second error region and not to the base mesh vertices in the first error region, wherein applying the non-rigid transformation changes at least one of a size or shape of the second error region;
(iv) iterate operations (i), (ii), and (iii) until a transformed base mesh corresponding to the base mesh transformed by operations (ii) and (iii) meets a convergence criterion; and
(v) in response to a determination that the convergence criterion is met, output a blendshape for the virtual avatar based at least in part on the transformed base mesh,

wherein the hardware processor is further programmed to:
determine a first falloff region;
determine a second falloff region outside of and enclosing the first falloff region, the second falloff region associated with a lower error than the first falloff region;
apply the rigid transformation or the non-rigid transformation for base mesh vertices in the first falloff region; and
feather the rigid transformation or the non-rigid transformation for base mesh vertices in the second falloff region.

2. The system of claim 1 , wherein the target mesh is determined from photographic scans of a human or animal subject performing a pose.

3. The system of claim 1 , wherein the plurality of error regions comprises a third error region having a third error that is less than the second error.

4. The system of claim 3 , wherein the hardware processor is programmed to not apply a transformation to the base mesh vertices in the third error region.

5. The system of claim 1 , wherein the rigid transformation comprises a rotation and a translation.

6. The system of claim 1 , wherein the rigid transformation comprises a rigid nearest-neighbor transformation.

7. The system of claim 1 , wherein the non-rigid transformation comprises a closest point on surface transformation.

8. The system of claim 1 , wherein the hardware processor is further programmed to not apply a transformation to base mesh vertices outside the second falloff region.

9. The system of claim 1 , wherein the hardware processor is programmed to determine a size of the first falloff region based at least partly on an error value associated with the first error region or the second error region.

10. The system of claim 1 , wherein the hardware processor is further programmed to remove one or more artifacts from the transformed base mesh after completion of operations (ii) and (iii).

11. The system of claim 1 , wherein the convergence criterion comprises a maximum number of iterations or an error between the transformed base mesh and the target mesh passing below a threshold error.

12. The system of claim 1 , wherein the hardware processor is further programmed to generate a heatmap based at least partly on the plurality of error regions.

13. A method for matching a base mesh for a virtual avatar to a target mesh of the virtual avatar, the method comprising:
by a hardware processor:
(i) determining a plurality of error regions between a target mesh for a virtual avatar and a base mesh for the virtual avatar, the target mesh comprising a plurality of target mesh vertices and the base mesh comprising a plurality of base mesh vertices, each error region representing a deviation in distance between positions of the base mesh vertices relative to positions of the target mesh vertices, the plurality of error regions comprising a first error region having a first error and a second error region having a second error, the second error less than the first error;
(ii) applying a rigid transformation to the base mesh vertices in the first error region, wherein applying the rigid transformation does not change a size and shape of the first error region;
(iii) applying a non-rigid transformation to the base mesh vertices in the second error region and not to the base mesh vertices in the first error region, wherein applying the non-rigid transformation changes at least one of a size or shape of the second error region;
(iv) iterating operations (i), (ii), and (iii) until a transformed base mesh corresponding to the base mesh transformed by operations (ii) and (iii) meets a convergence criterion; and
(v) in response to a determination that the convergence criterion is met, outputting a blendshape for the virtual avatar based at least in part on the transformed base mesh;

the method further comprising, by the hardware processor:
determining a first falloff region;
determining a second falloff region outside of and enclosing the first falloff region, the second falloff region associated with a lower error than the first falloff region;
applying the rigid transformation or the non-rigid transformation for base mesh vertices in the first falloff region; and
feathering the rigid transformation or the non-rigid transformation for base mesh vertices in the second falloff region.

14. The method of claim 13 , further comprising, by the hardware processor, not applying a transformation to base mesh vertices outside the second falloff region.

15. The method of claim 13 , further comprising, by the hardware processor, determining a size of the first falloff region based at least partly on an error value associated with the first error region or the second error region.

16. The method of claim 13 , wherein the convergence criterion comprises a maximum number of iterations or an error between the transformed base mesh and the target mesh passing below a threshold error.

Assignees

• Magic Leap Inc

Inventors

• Comer Sean Michael

Classifications

• B63B21/26 Primary

  securing to bed · CPC title

• G06T17/20 Primary

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• F01N2590/02

  for marine vessels or naval applications · CPC title

• B63H21/32

  Arrangements of propulsion power-unit exhaust uptakes; Funnels peculiar to vessels · CPC title

• B63B35/44

  Floating buildings, stores, drilling platforms, or workshops, e.g. carrying water-oil separating devices {(construction methods for floating offshore platforms B63B75/00)} · CPC title