Wearable device for facilitating enhanced interaction

Patent metadata

Field	Value
Publication number	US-12277361-B2
Application number	US-202318372659-A
Country	US
Kind code	B2
Filing date	Sep 25, 2023
Priority date	Jun 16, 2017
Publication date	Apr 15, 2025
Grant date	Apr 15, 2025

Abstract

Wearable head-mounted displays, such as virtual reality systems, present immersive experiences and environments to a wearer. However, the head-mounted displays, as well as the immersive environments that they produce, limit the wearer's ability to interact with outside observers. For example, a wearer may not be able to see outside observers, and outside observers may not have any insight to what the wearer is experiencing or where the wearer's attention is directed. Accordingly, a wearable electronic device may include an outward-facing display configured to display information to outside observers, such as images of the wearer's face or images that represent or indicate the state of the wearer and/or the head mounted display.

First claim

What is claimed is:

1. A head-mounted device comprising:
a housing;
a wearer-facing camera positioned in the housing and configured to capture a first image of a portion of a wearer's face;
an outward-facing display positioned in the housing and configured to display a second image of the portion of the wearer's face, the second image based on the captured first image;
an outward-facing sensing system configured to:
detect a lighting condition of an environment surrounding the head-mounted device; and
determine a position of an external observer relative to the outward-facing display, wherein the position of the external observer relative to the outward-facing display changes over time; and

a processor operatively coupled to the wearer-facing camera, the outward-facing display, and the outward-facing sensing system and configured to produce the second image of the portion of the wearer's face by:
modifying the captured first image based at least in part on the detected lighting condition of the environment surrounding the head-mounted device; and
repeatedly modifying the captured first image based at least in part on the position of the external observer relative to the outward-facing display as the position of the external observer relative to the outward-facing display changes over time.

2. The head-mounted device of claim 1 , wherein modifying the captured first image based at least in part on the detected lighting condition of the environment surrounding the head-mounted device comprises changing a brightness of the second image.

3. The head-mounted device of claim 1 , wherein:
the lighting condition is a first lighting condition; and
the outward-facing sensing system is further configured to detect a second lighting condition of the environment surrounding the head-mounted device, the second lighting condition different from the first lighting condition.

4. The head-mounted device of claim 3 , wherein:
the first lighting condition is a light intensity; and
the second lighting condition is a light direction.

5. The head-mounted device of claim 1 , wherein:
the lighting condition is light color; and
modifying the captured first image comprises changing a color of at least a portion of the first image.

6. The head-mounted device of claim 1 , wherein:
the first image of the portion of the wearer's face is a real-time video image of the wearer's eyes; and
the second image of the portion of the wearer's face is a modified real-time video image of the wearer's eyes.

7. The head-mounted device of claim 1 , wherein:
the lighting condition is a light direction; and
modifying the captured first image of the portion of the wearer's eyes comprises adding a shadow to the first image, the position of the shadow defined at least in part on the light direction.

8. The head-mounted device of claim 1 , wherein:
in a first mode of operation, the second image is displayed on the outward-facing display; and
in a second mode of operation, the outward-facing display is inactive.

9. The head-mounted device of claim 8 , wherein the processor is configured to:
in response to a determination that the external observer is present, operate the head-mounted device in the first mode of operation; and
in response to a determination that the external observer is not present, operate the head-mounted device in the second mode of operation.

10. A head-mounted display system comprising:
a first display configured to display content to a wearer;
a second display configured to display content away from the wearer;
a camera configured to capture a video image of the wearer's eyes;
a sensing system configured to:
detect a lighting condition of an environment surrounding the head-mounted display system; and
determine a position of the head-mounted display system relative to an external observer, the position of the head-mounted display system varying, relative to the external observer, over time;

a processor operatively coupled to the camera and the sensing system and configured to:
modify the video image of the wearer's eyes in real-time based at least in part on:
the lighting condition; and
the position of the head-mounted display system as the position of the head-mounted display system varies, relative to the external observer, over time; and

cause the modified video image to be displayed on the second display.

11. The head-mounted display system of claim 10 , wherein:
in a first mode of operation, the modified video image is displayed on the second display; and
in a second mode of operation, the second display is inactive.

12. The head-mounted display system of claim 11 , wherein the processor is configured to:
in response to a determination that the external observer is present, operate the head-mounted display system in the first mode of operation; and
in response to a determination that the external observer is not present, operate the head-mounted display system in the second mode of operation.

13. The head-mounted display system of claim 10 , wherein modifying the video image comprises changing a brightness of the video image.

14. The head-mounted display system of claim 10 , wherein modifying the video image comprises modifying the video image such that the video image of the wearer's eyes, as displayed on the second display, conforms to contours of the wearer's face.

15. The head-mounted display system of claim 10 , wherein:
the camera is a first camera;
the head-mounted display system further comprises a second camera configured to capture a video image of the external observer; and
the processor is further configured to display the video image of the external observer on the first display.

16. A head-mounted device comprising:
a housing;
a wearer-facing display positioned within the housing and configured to display at least partially virtual content to a wearer;
a wearer-facing sensing system positioned in the housing and configured to capture data corresponding to a facial expression of the wearer;
an outward-facing display positioned in the housing and configured to display a first image including a portion of the wearer's face;
an outward-facing sensing system configured to detect a lighting condition of an external environment and detect a varying position of an external observer relative to the wearer over time; and
a processor operatively coupled to the wearer-facing display, the wearer-facing sensing system, the outward-facing display, and the outward-facing sensing system and configured to:
produce the first image of the portion of the wearer's face by modifying a captured second image of the wearer based at least in part on the detected lighting condition and the facial expression of the wearer; and
in response to detecting the varying position of the external observer relative to the wearer, periodically modify the captured second image of the wearer based at least in part on the varying position of the external observer.

17. The head-mounted device of claim 16 , wherein detecting the lighting condition comprises detecting whether the lighting is diffuse or directional.

18. The head-mounted device of claim 17 , wherein in response to detecting that the lighting condition is directional, modifying the captured second image of the wearer comprises adding a highlight to the captured second image.

19. The head-mounted device of claim 16 , wherein:
detecting the lighting condition comprises detecting a light intensity of the external environment; and
in response to

Assignees

• Apple Inc

Inventors

• Ive Jonathan
• Hoenig Julian
• Jaede Julian
• Kim Seung Wook
• Wilson Chris
• Sorrentino Iii William A
• Dye Alan C
• Lemay Stephen O

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T3/18

  Image warping, e.g. rearranging pixels individually · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06V40/174

  Facial expression recognition · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title