Rendering of enrolled user's face for external display

Patent metadata

Field	Value
Publication number	US-2025182402-A1
Application number	US-202519043709-A
Country	US
Kind code	A1
Filing date	Feb 3, 2025
Priority date	Mar 31, 2023
Publication date	Jun 5, 2025
Grant date	—

Abstract

Various implementations disclosed herein include devices, systems, and methods that present a view of a device user's face portion, that would otherwise be blocked by an electronic device positioned in front of the face, on an outward-facing display of the user's device. The view of the user's face portion may be configured to enable observers to see the user's eyes and facial expressions as if they were seeing through a clear device at the user's actual eyes and facial expressions. Various techniques are used to provide views of the user's face that are realistic, that show the user's current facial appearance, and/or that present the face portion with 3D spatial accuracy, e.g., each eye appearing to be in its actual 3D position. Some implementations combine live data with previously-obtained data, e.g., combining live data with enrollment data.

First claim

What is claimed is:

1 . A method comprising:
at a wearable electronic device having a processor:
obtaining first data corresponding to an appearance of areas of a face portion comprising skin around the eyes of a user;
obtaining second data corresponding to a skin color of the user;
generating a representation of the face portion based on the first data and second data; and
presenting the representation of the face portion on an outward facing display of the wearable electronic device.

2 . The method of claim 1 , wherein the representation is based on a 3D mesh representing a 3D shape of the face portion.

3 . The method of claim 1 , wherein the representation is based on an orthographic projection depicting the face portion.

4 . The method of claim 1 , wherein the representation is based on a set of multiple orthographic projections depicting the face portion in different lighting conditions.

5 . The method of claim 4 , wherein the different lighting conditions correspond to illumination of the face from different primary light source directions.

6 . The method of claim 4 , wherein the orthographic projections depict the face portion with neutral eyes open and looking straight forward.

7 . The method of claim 1 , wherein skin color is not depicted in the first data.

8 . The method of claim 1 , wherein generating the representation of the face portion comprises inputting the first data and the second data to a neural network trained to output the representation of the face portion.

9 . The method of claim 8 , wherein generating the representation of the face portion comprises:
inputting, to a rendering process, eye camera data comprising depictions of a portion of the face portion as the first data; inputting, to the rendering process, a neutral 2D image depicting the face portion as the second data; and generating, the representation of the face portion via the rendering process.

10 . The method of claim 9 further comprising inputting neutral eye camera data to the rendering process.

11 . The method of claim 10 , wherein generating the representation of the face portion comprises determining a change based on the eye camera data and the neutral eye camera data.

12 . The method of claim 10 , wherein inputting the 2D image depicting the face portion comprises inputting a 2D image depicting the face portion in a lighting condition corresponding to a current lighting condition.

13 . The method of claim 12 , wherein inputting the 2D image depicting the face portion comprises selecting the 2D image depicting the face portion from a set of 2D images depicting the face portion in different lighting conditions, the selecting based on the current lighting condition.

14 . The method of claim 1 , wherein presenting the representation of the face portion on the outward facing display of the wearable electronic device comprises:
generating a 3D face portion representation by mapping the representation to a 3D representation of the face portion; and generating a view from a viewpoint of the 3D face portion representation from a viewpoint.

15 . The method of claim 1 , wherein the view is such that a viewer at the viewpoint observes the eyes and surrounding portions of the face portion in 3D positions corresponding to their actual 3D positions.

16 . The method of claim 1 further comprising providing different views of the face portion via a lenticular display, wherein the viewpoint is determined based on determining a relative position of an observer to the wearable electronic device.

17 . The method of claim 1 , wherein the wearable electronic device comprises a head mounted device (HMD) that blocks a direct view the face portion when the electronic device is being worn.

18 . A wearable device comprising:
a non-transitory computer-readable storage medium; one or more sensors; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising: obtaining first data corresponding to an appearance of areas of a face portion comprising skin around the eyes of a user; obtaining second data corresponding to a skin color of the user; generating a representation of the face portion based on the first data and second data; and presenting the representation of the face portion on an outward facing display of the wearable electronic device.

19 . A non-transitory computer-readable storage medium, storing program instructions executable on a device including one or more processors to perform operations comprising:
obtaining first data corresponding to an appearance of areas of a face portion comprising skin around the eyes of a user; obtaining second data corresponding to a skin color of the user; generating a representation of the face portion based on the first data and second data; and presenting the representation of the face portion on an outward facing display of the wearable electronic device.

Assignees

• Apple Inc

Inventors

• Cadet Gilles M
• Guan Shaobo
• Soares Olivier
• Fyffe Graham L
• Song Yang

Classifications

• G06F3/1431

  using a single graphics controller · CPC title

• G06T13/40

  of characters, e.g. humans, animals or virtual beings · CPC title

• G02B27/017 Primary

  Head mounted · CPC title

• G06V40/165

  using facial parts and geometric relationships · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title