Localizing Binaural Sound to Objects

Patent metadata

Field	Value
Publication number	US-2025220386-A1
Application number	US-202519086627-A
Country	US
Kind code	A1
Filing date	Mar 21, 2025
Priority date	Nov 17, 2016
Publication date	Jul 3, 2025
Grant date	—

Abstract

An electronic device identifies a location of a physical object that is away from a listener. The electronic device convolves sound so the sound externally localizes as binaural sound to the physical object. The sound plays to the listener through earphones so the listener hears the sound as emanating from the physical object.

First claim

1 .- 20 . (canceled)

21 . A method comprising:
displaying, with a head mounted display (HMD) worn on a head of a person, a virtual object that appears in a physical environment where the person is located; detecting, with one or more cameras or sensors in the HMD, a direction of gaze of the person looking at the virtual object being displayed; and changing a size of the virtual object in response to the HMD detecting the direction of gaze is looking at the virtual object being displayed.

22 . The method of claim 21 further comprising:
playing, with speakers in the HMD, a binaural sound that originates to the person where the virtual object is located in the physical environment in response to the HMD detecting the direction of gaze is looking at the virtual object being displayed.

23 . The method of claim 21 further comprising:
changing a color of the virtual object in response to the HMD detecting the direction of gaze is looking at the virtual object being displayed.

24 . The method of claim 21 further comprising:
blinking the virtual object in response to the HMD detecting the direction of gaze is looking at the virtual object being displayed.

25 . The method of claim 21 further comprising:
extracting, from the direction of gaze, coordinates of the virtual object in the physical environment; and
retrieving head-related transfer functions (HRTFs) with coordinates that correspond to the coordinates of the virtual object in the physical environment.

26 . The method of claim 21 further comprising:
capturing, with one or more of a camera or sensor in the HMD, images of the physical environment that is a room; and
generating, from the images of the physical environment, a three-dimensional (3D) model of the room; and
displaying, with the HMD, the 3D model of room while the user walks around the physical environment.

27 . The method of claim 21 further comprising:
assisting the person in locating a physical object in the physical environment when the physical object is located outside a field-of-view (FOV) of the person wearing the HMD by playing a binaural sound that localizes to where the physical object is located in the physical environment outside the FOV.

28 . The method of claim 21 further comprising:
displaying, with the HMD, the virtual object as an augmented reality (AR) image of an intelligent personal assistant (IPA) in the physical environment,
wherein the HMD includes electronic glasses, and the AR image of the IPA displays a virtual cue in response to the HMD detecting the direction of gaze is looking at the AR image of the IPA.

29 . The method of claim 21 further comprising:
playing, with speakers in the HMD, a binaural sound that originates to the person where the virtual object is located in the physical environment in response to the HMD detecting the person is within a predetermined distance from where the virtual object is located in the physical environment.

30 . A method comprising:
capturing, with one or more cameras in a head mounted display (HMD) worn on a head of a person, video of a room where the person is located; detecting, with one or more cameras or sensors in the HMD, a direction of gaze of the person looking at a virtual object displayed in the video of the room; and flashing or blinking the virtual object in response to the HMD detecting the direction of gaze of the person looking at the virtual object displayed in the video of the room.

31 . The method of claim 30 further comprising:
playing, with speakers in the HMD, binaural sound that localizes to the virtual object in response to the HMD detecting the direction of gaze of the person looking at the virtual object displayed in the video of the room.

32 . The method of claim 30 further comprising:
displaying the virtual object as an intelligent personal assistant (IPA); and
changing a size of the of the IPA displayed in the video of the room in response to the HMD detecting the direction of gaze of the person looking at the virtual object displayed in the video of the room.

33 . The method of claim 30 further comprising:
changing a shape of the virtual object in response to the HMD detecting the direction of gaze of the person looking at the virtual object displayed in the video of the room.

34 . The method of claim 30 further comprising:
assisting the person in locating the virtual object when the virtual object is located outside a field-of-view (FOV) of the person wearing the HMD by playing sound that localizes to where the virtual object is located outside the FOV.

35 . The method of claim 30 further comprising:
playing binaural sound that localizes to the virtual object in response to the person selecting the virtual object by looking at the virtual object.

36 . A head mounted display (HMD) comprising:
one or more cameras that capture video of a physical environment where a person wearing the HMD is located; a display that displays a virtual object in the video of the physical environment where the person wearing the HMD is located; one or more cameras that capture a direction of gaze of the person looking at the virtual object in the video of the physical environment where the person wearing the HMD is located; and speakers that play binaural sound that localizes to the virtual object in response to the person selecting the virtual object by looking at the virtual object in the video of the physical environment.

37 . The HMD of claim 36 , wherein the display highlights the virtual object in response to the direction of gaze of the person looking at the virtual object in the video of the physical environment.

38 . The HMD of claim 36 , wherein the HMD assists the person in locating the virtual object when the virtual object is located outside a field-of-view (FOV) of the person by playing a binaural sound that originates from the virtual object that is located outside the FOV.

39 . The HMD of claim 36 , wherein the virtual object is an intelligent personal assistant (IPA) displayed in the video of the physical environment, and the speakers automatically play a voice of the IPA in binaural sound that originates from the IPA in response to the direction of gaze of the person looking at the IPA in the video of the physical environment.

40 . The HMD of claim 36 , wherein the display flashes or blinks the virtual object in response to the direction of gaze of the person looking at the virtual object in the video of the physical environment.

Assignees

• Norris Glen A
• Lyren Philip Scott

Inventors

• Norris Glen A
• Lyren Philip Scott

Classifications

• H04S2400/01

  Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved · CPC title

• H04R5/04

  Circuit arrangements, {e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments (combinations of amplifiers H03F3/68; stereophonic systems H04S)} · CPC title

• H04S2400/15

  Aspects of sound capture and related signal processing for recording or reproduction · CPC title

• H04S2400/11

  Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence) · CPC title

• H04S7/40

  Visual indication of stereophonic sound image · CPC title