Periocular test for mixed reality calibration

Patent metadata

Field	Value
Publication number	US-12339458-B2
Application number	US-202318401020-A
Country	US
Kind code	B2
Filing date	Dec 29, 2023
Priority date	Oct 5, 2016
Publication date	Jun 24, 2025
Grant date	Jun 24, 2025

Abstract

A wearable device can include an inward-facing imaging system configured to acquire images of a user's periocular region. The wearable device can determine a relative position between the wearable device and the user's face based on the images acquired by the inward-facing imaging system. The relative position may be used to determine whether the user is wearing the wearable device, whether the wearable device fits the user, or whether an adjustment to a rendering location of a virtual object should be made to compensate for a deviation of the wearable device from its normal resting position.

First claim

What is claimed is:

1. A head-mounted device (HMD) comprising:
one or more inward-facing cameras; and
a hardware processor programmed to periodically adjust a rendering location of a virtual object to be presented by the HMD, including:
receiving one or more images of a periocular region of a face of a user of the HMD, the one or more images captured by the one or more inward-facing cameras;
calculating a plurality of fit parameters based on an appearance of the periocular region of the user in the one or more images, wherein the plurality of fit parameters include: i) at least one distance between the HMD and the face, and ii) an interpupillary distance (IPD) between pupils of a first eye and a second eye of the user;
determining a relative position between the HMD and the face based at least partly on the plurality of fit parameters;
adjusting the rendering location of the virtual object based, at least in part, on the relative position between the HMD and the face exceeding a predetermined threshold amount, including determining that the IPD that is calculated based on the appearance of the periocular region is outside a threshold range of acceptable IPD, wherein the threshold range of acceptable IPD is calculated based on sampled IPD measurements of a group of people that share at least one common characteristic with the user; and
causing the HMD to render the virtual object at the adjusted rendering location.

2. The HMD of claim 1 , wherein determining the relative position between the HMD and the face further includes tracking one or more periocular features in the periocular region using visual keypoints.

3. The HMD of claim 2 , wherein the visual keypoints are computed using at least one of: scale-invariant feature transform, speeded up robust features, oriented FAST and rotated BRIEF, binary robust invariant scalable keypoints, or fast retina keypoint.

4. The HMD of claim 1 , wherein determining the relative position between the HMD and the face further includes matching a region of the face with a dense map encoding at least a portion of the head of the user, and wherein the dense map comprises a respective three-dimensional shape of each of a plurality of regions of the face.

5. The HMD of claim 4 , wherein the matching of the region of the face with the dense map is calculated using iterative closest point algorithm.

6. The HMD of claim 1 , wherein determining the relative position includes determining a forward tilt of the HMD relative to the face based at least partly on the distance between the HMD and the face being greater than the threshold amount, and wherein the rendering location of the virtual object is further based on the determined forward tilt.

7. The HMD of claim 1 , wherein the hardware processor is further programmed to calculate an adjustment to a normal rendering position of the virtual object based at least partly on the relative position between the HMD and the face, and wherein the hardware processor is further programmed to determine the rendering location of the virtual object based, at least in part, on the adjustment to the normal rendering position.

8. The HMD of claim 7 , wherein the hardware processor is further programmed to determine a normal resting position of an eye of the user that is associated with a rendering viewpoint of the HMD, wherein the normal rendering position of the virtual object corresponds to the normal resting position of the eye.

9. The HMD of claim 7 , wherein to calculate the adjustment to the normal rendering position of the virtual object, the hardware processor is programmed to:
determine a shift with respect to the normal resting position based at least partly on the relative position between the HMD and the face; and
shift a coordinate associated with the rendering viewpoint of the HMD based at least partly on the shift with respect to the normal resting position.

10. The HMD of claim 1 , wherein the relative position between the HMD and the face comprises one or more of: a horizontal shift, a vertical shift, a depth shift, a tilt to a side, or the forward tilt with respect to the normal resting position.

11. The HMD of claim 1 , wherein determining the relative position between the HMD and the face comprises determining a first relative position between the HMD and the first eye of the user and determining a second relative position between the HMD and the second eye of the user.

12. The HMD of claim 11 , wherein the rendering location of the virtual object is adjusted based at least in part on determining that the first relative position is different from the second relative position.

13. The HMD of claim 12 , wherein adjusting the rendering location of the virtual object further comprises determining a first rendering location of the virtual object to be rendered for viewing by the first eye and determining a second rendering location of the virtual object to be rendered for viewing by the second eye.

14. The HMD of claim 1 , wherein the rendering location is periodically updated per half second, second, 10 seconds, 30 seconds, or one minute.

15. The HMD of claim 11 , wherein determining the relative position includes determining a forward tilt of the HMD relative to the face based at least partly on the distance between the HMD and the face being greater than the threshold amount and further based on the first relative position being approximately equal to the second relative position, and wherein the rendering location of the virtual object is further based on the determined forward tilt.

Assignees

• Magic Leap Inc

Inventors

• Kaehler Adrian
• Bradski Gary
• Badrinarayanan Vijay

Classifications

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06T3/20

  Linear translation of whole images or parts thereof, e.g. panning · CPC title

• G02B2027/0187

  slaved to motion of at least a part of the body of the user, e.g. head, eye · CPC title

• G02B2027/0181

  Adaptation to the pilot/driver · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title