Face model capture by a wearable device

Patent metadata

Field	Value
Publication number	US-12474588-B2
Application number	US-202418981592-A
Country	US
Kind code	B2
Filing date	Dec 15, 2024
Priority date	Sep 28, 2016
Publication date	Nov 18, 2025
Grant date	Nov 18, 2025

Abstract

Systems and methods for generating a face model for a user of a head-mounted device are disclosed. The head-mounted device can include one or more eye cameras configured to image the face of the user while the user is putting the device on or taking the device off. The images obtained by the eye cameras may be analyzed using a stereoscopic vision technique, a monocular vision technique, or a combination, to generate a face model for the user. The face model can be used to generate a virtual image of at least a portion of the user's face, for example to be presented as an avatar.

First claim

What is claimed is:

1 . A display system comprising:
a wearable display configured to present a three-dimensional (3D) environment to a user of the wearable display; an imaging system coupled to the wearable display, the imaging system including at least one inward-facing camera configured to capture images of at least a portion of a face of the user; and at least one processor communicatively coupled to the wearable display and the imaging system, the at least one processor configured to execute software to perform operations comprising:
analyzing the images captured by the at least one inward-facing camera to determine that at least a threshold number of consecutively captured images have substantially the same content;
determining a stopping trigger based on determining that at least the threshold number of consecutively captured images have substantially the same content;
generating a face model of at least the portion of the face of the user, based at least partly on further analyzing the images captured during a time period bounded by the stopping trigger; and
using the face model to generate, for presentation, a virtual image of at least the portion of the face.

2 . The display system of claim 1 , wherein the images used to generate the face model are captured while the at least one inward-facing camera is at a distance from the face that is greater than the distance between the at least one inward-facing camera and the face when the wearable display is being worn.

3 . The display system of claim 1 , wherein the display system further comprises at least one sensor, and wherein the time period is further bounded by a starting trigger that is determined based at least partly on data received from the at least one sensor.

4 . The display system of claim 3 , wherein the at least one sensor includes an inertial measurement unit, wherein the data describes an acceleration of the wearable display, and wherein the starting trigger is determined based at least partly on the acceleration exceeding a threshold acceleration.

5 . The display system of claim 1 , wherein analyzing the images to generate the face model comprises converting the one or more images to point clouds using a stereo vision algorithm.

6 . The display system of claim 5 , wherein analyzing the images to generate the face model further comprises combining at least two of the point clouds using an iterative closest point algorithm.

7 . The display system of claim 5 , wherein the stereo vision algorithm comprises at least one of a block-matching algorithm, a semi-global matching algorithm, a semi-global block-matching algorithm, or a neural network algorithm.

8 . The display system of claim 1 , wherein generating the face model comprises:
accessing a pre-existing face model; and updating the pre-existing face model based at least partly on the analyzing of the images.

9 . The display system of claim 8 , wherein the pre-existing face model comprises at least one of a generic face model or a previously generated face model of the user.

10 . The display system of claim 1 , wherein generating the face model further comprises:
accessing one or more images of the portion of the face previously acquired by at least one of the imaging system or another imaging device; wherein generating the face model is further based on analyzing the one or more accessed images.

11 . The display system of claim 1 , wherein determining the stopping trigger is further based on determining that at least the threshold number of consecutively captured images are captured within a threshold duration of time and have substantially the same content.

12 . The display system of claim 1 , wherein determining the stopping trigger is further based on determining that at least the threshold number of consecutively captured images each include at least one same feature of the face.

13 . The display system of claim 12 , wherein the at least one same feature includes at least one eye of the user.

14 . A computer-implemented method for generating a virtual image of at least a portion of a face of a user of a wearable display, the method performed by at least one processor executing instructions stored on non-transitory computer storage media, the method comprising:
analyzing images captured by at least one inward-facing camera coupled to the wearable display to determine that at least a threshold number of consecutively captured images have substantially the same content; determining a stopping trigger based on determining that at least the threshold number of consecutively captured images have substantially the same content; generating a face model of at least the portion of the face of the user, based at least partly on further analyzing the images captured during a time period bounded by the stopping trigger; and using the face model to generate, for presentation, the virtual image of at least the portion of the face.

15 . The computer-implemented method of claim 14 , wherein the images used to generate the face model are captured while the at least one inward-facing camera is at a distance from the face that is greater than the distance between the at least one inward-facing camera and the face when the wearable display is being worn.

16 . The computer-implemented method of claim 14 , wherein the time period is further bounded by a starting trigger that is determined based at least partly on data received from the at least one sensor.

17 . The computer-implemented method of claim 16 , wherein the at least one sensor includes an inertial measurement unit, wherein the data describes an acceleration of the wearable display, and wherein the starting trigger is determined based at least partly on the acceleration exceeding a threshold acceleration.

18 . The computer-implemented method of claim 14 , wherein determining the stopping trigger is further based on determining that at least the threshold number of consecutively captured images each include at least one same feature of the face.

19 . The computer-implemented method of claim 18 , wherein the at least one same feature includes at least one eye of the user.

20 . One or more non-transitory computer-readable storage media storing computer instructions which, when executed by at least one processor, instruct the processor to perform operations for generating a virtual image of at least a portion of a face of a user of a wearable display, the operations comprising:
analyzing images captured by at least one inward-facing camera coupled to the wearable display to determine that at least a threshold number of consecutively captured images have substantially the same content; determining a stopping trigger based on determining that at least the threshold number of consecutively captured images have substantially the same content; generating a face model of at least the portion of the face of the user, based at least partly on further analyzing the images captured during a time period bounded by the stopping trigger; and using the face model to generate, for presentation, the virtual image of at least the portion of the face.

Assignees

• Magic Leap Inc

Inventors

• Amayeh Gholamreza
• Kaehler Adrian
• Lee Douglas

Classifications

• G06V2201/12

  Acquisition of 3D measurements of objects · CPC title

• G06V40/19

  Sensors therefor · CPC title

• G06V40/169

  Holistic features and representations, i.e. based on the facial image taken as a whole · CPC title

• G06V20/64

  Three-dimensional [3D] objects · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title