Using ultrasound to improve IMU-based gesture detection

Patent metadata

Field	Value
Publication number	US-9811311-B2
Application number	US-201414230038-A
Country	US
Kind code	B2
Filing date	Mar 31, 2014
Priority date	Mar 17, 2014
Publication date	Nov 7, 2017
Grant date	Nov 7, 2017

Abstract

The present disclosure provides techniques for improving IMU-based gesture detection by a device using ultrasonic Doppler. A method may include detecting the onset of a gesture at a first device based on motion data obtained from an IMU of the first device. An indication of the detection of the onset of the gesture may be provided to a second device. Next, a first audio signal may be received from the second device. As a result, the gesture may be identified based on the motion data and the received first audio signal. In some cases, a first token encoded within the first audio signal may be decoded and the first token may be provided to a third coordinating device. A confirmation message may be received from the third coordinating device based on the first token provided and identifying the gesture may be further based on the confirmation message.

First claim

The invention claimed is:

1. A computer-implemented method comprising:
determining, based on accelerometer or gyroscope data generated by one or more sensors on a first computing device, that a user of the first computing device is likely to have begun performing a movement using the first computing device;
after determining that the user of the first computing device is likely to have begun performing the movement using the first computing device and before the first computing device has identified a particular gesture corresponding to the movement, transmitting, from the first computing device to a different, physically separate, second computing device, data indicating that the user of the first computing device is likely to have begun performing the movement using the first computing device;
before the first computing device has identified the particular gesture corresponding to the movement, obtaining, by the first computing device, audio data corresponding to a sound generated by the second computing device in response to the first computing device transmitting data to the different, physically separate, second computing device indicating that the user of the first computing device was likely to have begun performing the movement using the first computing device;
before the first computing device has identified the particular gesture corresponding to the movement, identifying one or more frequency shift peak values from the audio data; and
identifying, by the first computing device, the particular gesture corresponding to the movement based at least on (i) the accelerometer or gyroscope data that was used to determine that the user of the first computing device was likely to have begun performing the movement using the first computing device, and (ii) the one or more of the frequency shift peak values that are identified from the audio data that (I) corresponds to the sound generated by the second computing device and (II) was obtained in response to the first computing device transmitting data to the different, physically separate, second computing device indicating that the user of the first computing device was likely to have begun performing the movement using the first computing device.

2. The computer-implemented method of claim 1 , further comprising:
executing a function corresponding to the particular gesture; and
determining a direction associated with the particular gesture.

3. The computer-implemented method of claim 1 , wherein the sound generated by the second computing device is not audible to the user.

4. The computer-implemented method of claim 1 , wherein identifying, by the first computing device, the particular gesture corresponding to the movement based at least on (i) the accelerometer or gyroscope data that was used to determine that the user of the first computing device was likely to have begun performing the movement using the first computing device, and (ii) the one or more of the frequency shift peak values that are identified from the audio data that (I) corresponds to the sound generated by the second computing device and (II) was obtained in response to the first computing device transmitting data to the different, physically separate, second computing device indicating that the user of the first computing device was likely to have begun performing the movement using the first computing device comprises:
receiving an indication of an expected audio signal, the expected audio signal having a particular frequency and one or more particular frequency peaks;
comparing the obtained audio data with the indication of the expected audio signal to determine one or more peak offsets; and
identifying the particular gesture corresponding to the movement based on the comparison of the obtained audio data with the indication of the expected audio signal and the determined one or more peak offsets.

5. The computer-implemented method of claim 1 , wherein obtaining, by the first computing device, audio data corresponding to the sound generated by the second computing device in response to the first computing device transmitting data to the different, physically separate, second computing device indicating that the user of the first computing device was likely to have begun performing the movement using the first computing device comprises receiving the audio data from a third computing device; and
wherein the audio data corresponding to the sound generated by the second computing device includes a token.

6. The computer-implemented method of claim 1 , wherein identifying the one or more frequency shift peak values from the audio data comprises determining a velocity of the first computing device relative to the second computing device.

7. The computer-implemented method of claim 1 , wherein:
the accelerometer or gyroscope data includes an angle associated with a yaw of the first computing device, an angle associated with a pitch of the first computing device, and an angle associated with a roll of the first computing device; and
transmitting, from the first computing device to the different, physically separate, second computing device, the data indicating that the user of the first computing device is likely to have begun performing the movement using the first computing device comprises:
transmitting, to the second computing device, data indicating a change in one or more of the yaw, the pitch, and the roll of the first computing device.

8. The computer-implemented method of claim 1 , wherein the accelerometer or gyroscope data includes an angle associated with a yaw of the first computing device, an angle associated with a pitch of the first computing device, and an angle associated with a roll of the first computing device, and does not include the sound data or data derived from sound data;
wherein transmitting, from the first computing device to the different, physically separate, second computing device, the data indicating that the user of the first computing device is likely to have begun performing the movement using the first computing device comprises transmitting, to the second computing device, data indicating a change in one or more of the yaw, the pitch, and the roll of the first computing device; and
wherein identifying, by the first computing device, a particular gesture corresponding to the movement based at least on (i) the accelerometer or gyroscope data that was used to determine that the user of the first computing device was likely to have begun performing the movement using the first computing device, and (ii) the one or more of the frequency shift peak values that are identified from the audio data that (I) corresponds to the sound generated by the second computing device and (II) was obtained in response to the first computing device transmitting data to the different, physically separate, second computing device indicating that the user of the first computing device was likely to have begun performing the movement using the first computing device comprises:
combining (i) the accelerometer or gyroscope data and (ii) the one or more of the frequency shift peak values;
generating data indicative of the particular gesture corresponding to the movement based on a combination of (i) the accelerometer or gyroscope data and (ii) the one or more of the frequency shift peak values that are identified from the audio data that corresponds to the sound generated by the second computing device; and
determining the particular gesture corresponding to the movement based on the data indicative of the particular gesture.

9. The computer-implemented method of claim 8 , wherein combining (i) the accelerometer or gyroscope data and (ii) the one or more of the frequency shift peak values comprises:

Assignees

• Google Inc

Inventors

• Smus Boris
• Plagemann Christian
• Mohan Ankit
• Rifkin Ryan Michael

Classifications

• G06F3/167 Primary

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G01S1/763

  using the Doppler shift introduced by the relative motion between beacon and receiver · CPC title

• G01S11/14

  using ultrasonic, sonic or infrasonic waves · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title