Guiding computational perception through a shared auditory space

Patent metadata

Field	Value
Publication number	US-9301722-B1
Application number	US-201414171662-A
Country	US
Kind code	B1
Filing date	Feb 3, 2014
Priority date	Feb 3, 2014
Publication date	Apr 5, 2016
Grant date	Apr 5, 2016

Abstract

In an example, a computer-implemented method receives one or more user inputs and captures a sound associated with a sound source via one or more capturing devices using sound source localization. The method then estimates one or more first posterior likelihoods of one or more positions of the sound source based on the one or more user inputs and a second posterior likelihood of a position of the sound source based on the sound. The method then estimates an overall posterior likelihood of an actual position of the sound source based on 1) the one or more first posterior likelihoods of the one or more positions of the sound source estimated based on the one or more user inputs and 2) the second posterior likelihood of the position of the sound source estimated based on the sound.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving, using one or more computing devices, one or more user inputs;
capturing, using the one or more computing devices, a sound associated with a sound source via one or more capturing devices using sound source localization;
estimating, using the one or more computing devices, one or more first posterior likelihoods of one or more positions of the sound source based on the one or more user inputs;
estimating, using the one or more computing devices, a second posterior likelihood of a position of the sound source based on the sound; and
estimating, using the one or more computing devices, an overall posterior likelihood of an actual position of the sound source based on 1) the one or more first posterior likelihoods of the one or more positions of the sound source estimated based on the one or more user inputs and 2) the second posterior likelihood of the position of the sound source estimated based on the sound.

2. The computer-implemented method of claim 1 , wherein estimating the overall posterior likelihood further includes:
fusing, using the one or more computing devices, the one or more first posterior likelihoods of the one or more positions of the sound source and the second posterior likelihood of the position of the sound source to produce the overall posterior likelihood.

3. The computer-implemented method of claim 2 , wherein the one or more first posterior likelihoods include two or more posterior likelihoods estimated based on two or more user inputs, and fusing the two or more first posterior likelihoods and the second posterior likelihood includes:
combining, using the one or more computing devices, the two or more first posterior likelihoods into a combined posterior likelihood;
scaling, using the one or more computing devices, the combined posterior likelihood; and
combining the scaled combined posterior likelihood with the second posterior likelihood.

4. The computer-implemented method of claim 1 , wherein the one or more user inputs include one or more of a user gesture, a user speech segment, and a user body pose.

5. The computer-implemented method of claim 1 , wherein
the one or more user inputs include a user gesture and a user body pose,
receiving the one or more user inputs further includes determining a gesturing direction from the user gesture and determining one or more of a position and orientation from the user body pose, and
estimating the one or more first posterior likelihoods further includes estimating a gesture-pose-based posterior likelihood of a position of the sound source based on the gesturing direction and the one or more of the position and orientation associated with the user body pose.

6. The computer-implemented method of claim 1 , wherein
the one or more inputs include a user speech segment and a user body pose;
receiving the one or more user inputs further includes determining a direction from the user speech segment and determining one or more of a position and orientation from the user body pose, and
estimating the one or more first posterior likelihoods further includes estimating a speech-pose-based posterior likelihood of a position of the sound source based on the direction associated with the speech segment and the one or more of the position and orientation associated with the user body pose.

7. The computer-implemented method of claim 1 , wherein
estimating the one or more first posterior likelihoods further includes generating one or more first evidence grids of likely sound source positions based on the one or more user inputs, the one or more first evidence grids reflecting the one or more first posterior likelihoods, respectively,
estimating the second posterior likelihood further includes generating a second evidence grid of likely sound source positions based on the sound, the second evidence grid reflecting the second posterior likelihood, and
estimating the overall posterior likelihood of the actual position of the sound source further includes combining the one or more first evidence grids and the second evidence grid.

8. The computer-implemented method of claim 1 , further comprising:
guiding, using the one or more computing devices, a mobile computing device to the actual position of the sound source.

9. The computer-implemented method of claim 1 , wherein the one or more capturing devices includes one or more of an image capturing device, a video capturing device, and an audio capturing device.

10. A computer program product comprising a non-transitory computer-readable medium storing a computer-readable program, wherein the computer-readable program, when executed on one or more computing devices, causes the one or more computing devices to:
receive one or more user inputs;
capture a sound associated with a sound source via one or more capturing devices using sound source localization;
estimate one or more first posterior likelihoods of one or more positions of the sound source based on the one or more user inputs;
estimate a second posterior likelihood of a position of the sound source based on the sound; and
estimate an overall posterior likelihood of an actual position of the sound source based on 1) the one or more first posterior likelihoods of the one or more positions of the sound source estimated based on the one or more user inputs and 2) the second posterior likelihood of the position of the sound source estimated based on the sound.

11. The computer program product of claim 10 , wherein to estimate the overall posterior likelihood further includes:
fusing the one or more first posterior likelihoods of the one or more positions of the sound source and the second posterior likelihood of the position of the sound source to produce the overall posterior likelihood.

12. The computer program product of claim 11 , wherein the one or more first posterior likelihoods include two or more posterior likelihoods estimated based on two or more user inputs, and to fuse the two or more first posterior likelihoods and the second posterior likelihood includes:
combining the two or more first posterior likelihoods into a combined posterior likelihood;
scaling the combined posterior likelihood; and
combining the scaled combined posterior likelihood with the second posterior likelihood.

13. The computer program product of claim 10 , wherein the one or more user inputs include one or more of a user gesture, a user speech segment, and a user body pose.

14. The computer program product of claim 10 , wherein
the one or more user inputs include a user gesture and a user body pose,
to receive the one or more user inputs further includes determining a gesturing direction from the user gesture and determining one or more of a position and orientation from the user body pose, and
to estimate the one or more first posterior likelihoods further includes estimating a gesture-pose-based posterior likelihood of a position of the sound source based on the gesturing direction and the one or more of the position and orientation associated with the user body pose.

15. The computer program product of claim 10 , wherein
the one or more inputs include a user speech segment and a user body pose;
to receive the one or more user inputs further includes determining a direction from the user speech segment and determining one or more of a position and orientation from the user body pose, and
to estimate the one or more first posterior likelihoods further includes estimating a speech-pose-based posterior likelihood of a position of the sound source based on the directio

Assignees

• Toyota Motor Co Ltd

Inventors

• Martinson Eric

Classifications

• G06F2218/22

  Source localisation; Inverse modelling · CPC title

• G06F18/00 Primary

  Pattern recognition · CPC title

• A61B5/7405 Primary

  using sound · CPC title