Rendering of a guest user's face for external display

Patent metadata

Field	Value
Publication number	US-2025054240-A1
Application number	US-202418931135-A
Country	US
Kind code	A1
Filing date	Oct 30, 2024
Priority date	Mar 31, 2023
Publication date	Feb 13, 2025
Grant date	—

Abstract

Various implementations disclosed herein include devices, systems, and methods that present a view of a device user's face portion, that would otherwise be blocked by an electronic device positioned in front of the face, on an outward-facing display of the user's device. The view of the user's face portion may be configured to enable observers to see the user's eyes and facial expressions as if they were seeing through a clear device at the user's actual eyes and facial expressions. Various techniques are used to provide views of the user's face that are realistic, that show the user's current facial appearance, and/or that present the face portion with 3D spatial accuracy, e.g., each eye appearing to be in its actual 3D position.

First claim

What is claimed is:

1 . A method comprising:
at a wearable electronic device having a processor: obtaining first data corresponding to a current appearance of a face portion of a user, the first data based on first sensor data captured via one or more sensors; determining that the user is different than another user associated with a user appearance enrollment for the device; in accordance with determining that the user is different than the another user associated with a user appearance enrollment for the device, determining to use second data corresponding to a characteristic of the user; obtaining the second data corresponding to the characteristic of the user; generating a representation of the face portion based on the first data and the second data; and presenting the representation of the face portion on an outward-facing display of the wearable electronic device.

2 . The method of claim 1 , wherein the characteristic of the user corresponds to skin of the user.

3 . The method of claim 1 , wherein the characteristic of the user is based on one or more images of hands of the user.

4 . The method of claim 3 , wherein the one or more images of the hands are captured during a hand enrollment.

5 . The method of claim 1 , wherein the characteristic of the user corresponds to a portion of the face portion not depicted in the first sensor data.

6 . The method of claim 1 , wherein generating the representation of the face portion comprises inputting the first data and the second data to a neural network trained to output a 3D mesh representing a shape of the face portion and an image representing a texture of the face portion.

7 . The method of claim 6 further comprising inputting live eye camera data and neutral eye camera data to the neural network.

8 . The method of claim 7 , wherein generating the representation of the face portion comprises determining a change based on the live eye camera data and the neutral eye camera data.

9 . The method of claim 6 , wherein presenting the representation of the face portion on the outward-facing display of the wearable electronic device comprises:
generating a live 3D face portion representation by mapping the image to the 3D mesh; and generating a view of the live 3D face portion representation from a viewpoint.

10 . The method of claim 9 , wherein the view depicts the user such that a viewer at the viewpoint observes the eyes and surrounding portions of the face portion in 3D positions corresponding to their actual 3D positions.

11 . The method of claim 9 , wherein the viewpoint is determined based on determining a relative position of an observer to the wearable electronic device.

12 . The method of claim 1 further comprising providing different views of the face portion over time based on a changing position of an observer, wherein the wearable electronic device provides the views via a lenticular display.

13 . The method of claim 1 , wherein the wearable electronic device comprises a head mounted device (HMD) that blocks a direct view the face portion when the wearable electronic device is being worn.

14 . A wearable electronic device comprising:
a non-transitory computer-readable storage medium; one or more sensors; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising: obtaining first data corresponding to a current appearance of a face portion of a user, the first data based on first sensor data captured via one or more sensors; determining that the user is different than another user associated with a user appearance enrollment for the device; in accordance with determining that the user is different than the another user associated with a user appearance enrollment for the device, determining to use second data corresponding to a characteristic of the user; obtaining the second data corresponding to the characteristic of the user; generating a representation of the face portion based on the first data and the second data; and presenting the representation of the face portion on an outward-facing display of the wearable electronic device.

15 . The system of claim 14 , wherein the characteristic of the user corresponds to skin of the user based on one or more images of hands of the user captured during a hand enrollment.

16 . The system of claim 14 , wherein the characteristic of the user corresponds to a portion of the face portion not depicted in the first sensor data.

17 . The system of claim 14 , wherein generating the representation of the face portion comprises inputting the first data and the second data to a neural network trained to output a 3D mesh representing a shape of the face portion and an image representing a texture of the face portion.

18 . The system of claim 17 further comprising inputting live eye camera data and neutral eye camera data to the neural network.

19 . The system of claim 18 , wherein generating the representation of the face portion comprises determining a change based on the live eye camera data and the neutral eye camera data.

20 . The system of claim 17 , wherein presenting the representation of the face portion on the outward-facing display of the wearable electronic device comprises:
generating a live 3D face portion representation by mapping the image to the 3D mesh; and generating a view of the live 3D face portion representation from a viewpoint.

21 . The system of claim 20 , wherein the view depicts the user such that a viewer at the viewpoint observes the eyes and surrounding portions of the face portion in 3D positions corresponding to their actual 3D positions.

22 . The system of claim 20 , wherein the viewpoint is determined based on determining a relative position of an observer to the wearable electronic device.

23 . The system of claim 14 , wherein the operations further comprise providing different views of the face portion over time based on a changing position of an observer, wherein the wearable electronic device provides the views via a lenticular display.

24 . The system of claim 14 , wherein the wearable electronic device comprises a head mounted device (HMD) that blocks a direct view the face portion when the wearable electronic device is being worn.

25 . A non-transitory computer-readable storage medium, storing program instructions executable on a device including one or more processors to perform operations comprising:
obtaining first data corresponding to a current appearance of a face portion of a user, the first data based on first sensor data captured via one or more sensors; determining that the user is different than another user associated with a user appearance enrollment for the device; in accordance with determining that the user is different than the another user associated with a user appearance enrollment for the device, determining to use second data corresponding to a characteristic of the user; obtaining the second data corresponding to the characteristic of the user; generating a representation of the face portion based on the first data and the second data; and presenting the representation of the face portion on an outward-facing display of the wearable electronic devi

Assignees

• Apple Inc

Inventors

• Cadet Gilles M
• Soares Olivier
• Fyffe Graham L
• Song Yang
• Guan Shaobo

Classifications

• G06F3/1431 Primary

  using a single graphics controller · CPC title

• G06T13/40

  of characters, e.g. humans, animals or virtual beings · CPC title

• G02B27/017

  Head mounted · CPC title

• G06V40/165

  using facial parts and geometric relationships · CPC title

• G06T17/20 Primary

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title