Head-mountable device with indicators

Patent metadata

Field	Value
Publication number	US-11740316-B2
Application number	US-202016935052-A
Country	US
Kind code	B2
Filing date	Jul 21, 2020
Priority date	Aug 20, 2019
Publication date	Aug 29, 2023
Grant date	Aug 29, 2023

Abstract

A head-mountable device can include multiple microphones for directional audio detection. The head-mountable device can also include a speaker for audio output and/or a display for visual output. The head-mountable device can be configured to provide visual outputs based on audio inputs by displaying an indicator on a display based on a location of a source of a sound. The head-mountable device can be configured to audio outputs based on audio inputs by modifying an audio output of the speaker based on a detected sound and a target characteristic. Such characteristics can be based on a direction of a gaze of the user, as detected by an eye sensor.

First claim

What is claimed is:

1. A head-mountable device comprising:
multiple microphones;
a display;
an eye sensor;
a speaker; and
a controller configured to perform operations comprising:
detecting a sound with the microphones;
determining a location of a source of the sound with respect to the head-mountable device;
with the eye sensor, detecting a direction of an eye gaze;
determining when the location of the source is in the direction of the eye gaze;
when the location of the source is in the direction of the eye gaze, amplifying the sound with the speaker; and
while the sound is detected with the microphones and when the location of the source is outside a field-of-view provided by the display, displaying an indicator on the display based on the location of the source.

2. The head-mountable device of claim 1 , wherein, when the location of the source is within the field-of-view provided by the display, the indicator is displayed at a portion of the display that corresponds to the location of the source.

3. The head-mountable device of claim 1 , wherein the indicator is displayed at an edge of the display that corresponds to a direction of the location of the source relative to a center of the display.

4. The head-mountable device of claim 1 , wherein determining the location of the source of the sound with respect to the head-mountable device comprises determining which one of the microphones is most closely directed toward the location of the source.

5. The head-mountable device of claim 1 , wherein:
the controller is further configured to determine a target characteristic; and
determining the location of the source of the sound with respect to the head-mountable device is based on the sound and the target characteristic.

6. The head-mountable device of claim 5 , wherein the target characteristic is based on a user input that identifies the sound.

7. The head-mountable device of claim 1 , wherein the indicator is based on a characteristic of the source of the sound.

8. The head-mountable device of claim 1 , further comprising a camera, wherein the display is configured to display a view captured by the camera.

9. A head-mountable device comprising:
multiple microphones;
a camera;
a display configured to display a field-of-view captured by the camera;
an eye sensor configured to detect a direction of an eye gaze;
a speaker; and
a controller configured to perform operations comprising:
detecting a sound with the microphones;
determining whether a source of the sound is located outside the field-of-view of the camera; and
with the eye sensor and when the location of the source is inside the field-of-view provided by the display, detecting when the source of the sound is in the direction of the eye gaze;
when the location of the source is in the direction of the eye gaze, amplifying the sound with the speaker; and
while the sound is detected with the microphones and when the source is located outside the field-of-view of the camera, displaying an indicator on the display based on a location of the source.

10. The head-mountable device of claim 9 , wherein, when the location of the source is within the field-of-view of the camera, the indicator is displayed at a portion of the display that corresponds to the location of the source.

11. The head-mountable device of claim 9 , wherein when the source is located outside the field-of-view of the camera, the indicator is displayed at an edge of the display that corresponds to a direction of the location of the source relative to a center of the display.

12. The head-mountable device of claim 9 , wherein determining the location of the source of the sound with respect to the head-mountable device comprises determining which one of the microphones is most closely directed toward the location of the source.

13. The head-mountable device of claim 9 , wherein:
the controller is further configured to determine a target characteristic; and
determining the location of the source of the sound with respect to the head-mountable device is based on the sound and the target characteristic.

14. The head-mountable device of claim 13 , wherein the target characteristic is based on a user input that identifies the sound.

15. The head-mountable device of claim 9 , wherein the indicator is based on a characteristic of the source of the sound.

16. A head-mountable device comprising:
multiple microphones;
a display;
an eye sensor configured to detect a direction of an eye gaze;
a speaker; and
a controller configured to perform operations comprising:
detecting a sound with the microphones;
determining whether a source of the sound is located outside a field-of-view provided by the display;
while the sound is detected with the microphones and when the location of the source is within the field-of-view provided by the display, outputting a visual indicator at a portion of the display that corresponds to the location of the source and detecting with the eye sensor when the source of the sound is in the direction of the eye gaze; and
when the location of the source is in the direction of the eye gaze, amplifying the sound with the speaker; and
while the sound is detected with the microphones and when the location of the source is outside the field-of-view provided by the display, outputting the visual indicator at an edge of the display that corresponds to a direction of the location of the source relative to a center of the display.

17. The head-mountable device of claim 16 , wherein determining the location of the source of the sound with respect to the head-mountable device comprises determining which one of the microphones is most closely directed toward the location of the source.

18. The head-mountable device of claim 16 , wherein:
the controller is further configured to determine a target characteristic; and
determining the location of the source of the sound with respect to the head-mountable device is based on the sound and the target characteristic.

19. The head-mountable device of claim 18 , wherein the target characteristic is based on a user input that identifies the sound.

20. The head-mountable device of claim 16 , further comprising a camera, wherein the field-of-view provided by the display is based on a view captured by the camera.

Assignees

• Apple Inc

Inventors

• Poore Killian J
• Dey Stephen E
• Ness Trevor J

Classifications

• G01S3/8083 Primary

  determining direction of source · CPC title

• G02B27/0176

  characterised by mechanical features · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• H04R1/403 Primary

  loud-speakers · CPC title