Gesture detection based on information from multiple types of sensors

Patent metadata

Field	Value
Publication number	US-9389690-B2
Application number	US-201313767698-A
Country	US
Kind code	B2
Filing date	Feb 14, 2013
Priority date	Mar 1, 2012
Publication date	Jul 12, 2016
Grant date	Jul 12, 2016

Abstract

A method includes receiving a first output from a first sensor of an electronic device and receiving a second output from a second sensor of the electronic device. The first sensor has a first sensor type and the second sensor has a second sensor type that is different from the first sensor type. The method also includes detecting a gesture based on the first output and the second output according to a complementary voting scheme that is at least partially based on gesture complexity.

First claim

What is claimed is:

1. An apparatus comprising:
a first sensor configured to generate a first output;
a camera configured to generate a second output; and
a processor configured to:
detect a gesture based on at least one of the first output and the second output when a lighting level is greater than or equal to a threshold; and
deactivate at least a portion of the camera and perform gesture recognition based on the first output from the first sensor when the lighting level is less than the threshold.

2. The apparatus of claim 1 , wherein the first sensor comprises an ultrasound sensor.

3. The apparatus of claim 2 , further comprising:
an ultrasound processing path configured to process the first output; and
an image processing path configured to process the second output,
wherein the ultrasound processing path is configured to send gesture range information to the image processing path,
wherein the gesture range information includes data related to a position of an object relative to the ultrasound sensor,
wherein the camera is configured to identify an area of interest based on the position of the object, and
wherein the camera is configured to activate a portion of a sensor array of the camera that corresponds to the area of interest and to deactivate other portions of the sensor array of the camera.

4. The apparatus of claim 1 , further comprising a light detector internal to the camera and configured to determine the lighting level.

5. The apparatus of claim 1 , further comprising a light detector external to the camera and configured to determine the lighting level.

6. The apparatus of claim 1 , wherein
the processor is configured to perform gesture recognition by using a voting scheme to select between the first output and the second output conditioned on a gesture range exceeding a nearness threshold and being less than a farness threshold.

7. The apparatus of claim 6 , wherein the voting scheme is at least partially based on gesture complexity.

8. A method comprising:
receiving a first output from a first sensor of an electronic device, wherein the first sensor has a first sensor type;
receiving a second output from a second sensor of the electronic device, wherein the second sensor has a second sensor type that is different from the first sensor type, and wherein the second sensor type comprises a camera sensor type;
when a lighting level is greater than or equal to a threshold, detecting a gesture based on the first output and the second output according to a voting scheme that is at least partially based on gesture complexity; and
when the lighting level is less than the threshold, deactivating at least a portion of the second sensor and performing gesture recognition based on the first output from the first sensor.

9. The method of claim 8 , wherein the first sensor type comprises at least one of an ultrasound sensor type, an infrared sensor type, and a magnetic sensor type.

10. The method of claim 8 , wherein the first output identifies a first gesture with a first confidence score and wherein the second output identifies a second gesture with a second confidence score.

11. The method of claim 10 , wherein detecting the gesture according to the voting scheme comprises:
when the first sensor and the second sensor identify the same gesture, selecting the same gesture as an output gesture; and
when the first sensor and the second sensor identify different gestures, selecting a gesture having a higher confidence score as the output gesture.

12. The method of claim 11 , wherein detecting the gesture according to the voting scheme further comprises, when the first sensor and the second sensor identify gestures of varying complexity, selecting a more complex gesture as the output gesture.

13. The method of claim 11 , wherein the first sensor and the second sensor output data in accordance with a common data model.

14. The method of claim 13 , wherein a first processing path associated with the first sensor receives the data output by the second sensor, and wherein a second processing path associated with the second sensor receives the data output by the first sensor.

15. The method of claim 8 , further comprising receiving third output from a third sensor, wherein the gesture is detected further based on the third output from the third sensor.

16. An apparatus, comprising:
first sensor means for generating a first output and having a first sensor type;
second sensor means for generating a second output and having a second sensor type that is different from the first sensor type;
means for detecting a lighting level;
means for selectively deactivating the second sensor means in response to determining that the lighting level is less than a threshold; and
means for detecting a gesture, the means for detecting the gesture configured to, when the lighting level is greater than or equal to the threshold, perform gesture recognition based on the first output and the second output according to a voting scheme that is at least partially based on gesture complexity and configured to perform gesture recognition based on the first output from the first sensor means when the lighting level is less than the threshold.

17. The apparatus of claim 16 , further comprising
means for determining a lighting level; and means for determining a gesture range.

18. The apparatus of claim 17 , wherein the means for selectively deactivating is configured to selectively deactivate the at least a portion of the second sensor means further based on and the gesture range.

19. An apparatus, comprising:
an ultrasound sensor configured to generate a first output in accordance with a common data model and to provide the first output to an ultrasound processing path;
a camera configured to generate a second output in accordance with the common data model and to provide the second output to an image processing path;
a processor; and
a gesture detection module executable by the processor to detect a gesture based on at least one of the first output and the second output,
wherein the ultrasound processing path and the image processing path are configured to exchange data in accordance with the common data model, wherein the data includes a position of an object relative to the ultrasound sensor or the camera, and wherein the camera is configured to identify an area of interest based on the position of the object.

20. The apparatus of claim 19 , wherein the data exchanged in accordance with the common data model further includes the first output from the ultrasound sensor, the second output from the camera, data related to a range of the object to the ultrasound sensor or the camera, data associated with an active configuration of the ultrasound sensor or the camera, or any combination thereof.

21. The apparatus of claim 19 , wherein the camera is configured to activate a portion of a sensor array in the camera that corresponds to the area of interest and to deactivate other portions of the sensor array in the camera.

22. The apparatus of claim 19 , wherein the ultrasound processing path is configured to determine whether a particular detected gesture is a false positive based on the data received from the image processing path.

23. The apparatus of claim 19 , further comprising a light detector configured to detect a lighting level.

24. An apparatus comprising:
an ultrasound sensor configured to provide a first output to an ultras

Assignees

• Qualcomm Inc

Inventors

• Li Ren
• Clarkson Ian
• Gupta Samir K
• Krulce Darrell L

Classifications

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F1/3287

  by switching off individual functional units in the computer system · CPC title

• Y02B60/32

  Cross-Sectional Technologies · mapped topic

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title