Radar based user interface

Patent metadata

Field	Value
Publication number	US-10481696-B2
Application number	US-201615060545-A
Country	US
Kind code	B2
Filing date	Mar 3, 2016
Priority date	Mar 3, 2015
Publication date	Nov 19, 2019
Grant date	Nov 19, 2019

Abstract

An apparatus and method for radar based gesture detection. The apparatus includes a processing element and a transmitter configured to transmit radar signals. The transmitter is coupled to the processing element. The apparatus further includes a plurality of receivers configured to receive radar signal reflections, where the plurality of receivers is coupled to the processing element. The transmitter and plurality of receivers are configured for short range radar and the processing element is configured to detect a hand gesture based on the radar signal reflections received by the plurality of receivers.

First claim

What is claimed is:

1. An apparatus for gesture sensing comprising:
a processing element;
a depth sensor communicatively coupled to the processing element;
a color sensor;
a transmitter configured to transmit radar signals, wherein the transmitter is coupled to the processing element;
a plurality of receivers configured to receive radar signal reflections, wherein the plurality of receivers is coupled to the processing element, and
the processing element is configured to:
cross-calibrate the color sensor, the plurality of receivers, and the depth sensor;
generate location and velocity data based on the radar signal reflections received by the plurality of receivers and by performing a transform over time process;
register information received from the plurality of receivers with the information received from the depth sensor based on a cross-calibration result; and
determine a hand gesture based on the location and velocity data;

wherein the transmitter and plurality of receivers are configured for short range radar based detection, and
wherein the location and velocity data are generated further based on information generated by the depth sensor and information generated by the color sensor.

2. The apparatus as described in claim 1 , wherein the short range radar based detection has a range of one meter.

3. The apparatus as described in claim 1 , wherein the transmitter and plurality of receivers are a portion of a user interface device of a vehicle.

4. The apparatus as described in claim 1 , wherein a first pair of receivers of the plurality of receivers is configured to measure a vertical component of a gesture movement and a second pair of receivers of the plurality of receivers is configured to measure a horizontal component of a gesture movement.

5. The apparatus as described in claim 1 , wherein the processing element is configured to determine a 3D position and radial velocity for a signal reflection received by the plurality of receivers.

6. The apparatus as described in claim 5 , wherein the processing element is further configured to determine an elevation angle and an azimuth angle for an object from which a signal reflection received by the plurality of receivers.

7. The apparatus as described in claim 1 , wherein the transmitter and the plurality of receivers are operated in a frequency-modulated continuous-wave (FMCW) mode.

8. The apparatus as described in claim 1 , wherein the transmitter and the plurality of receivers are part of a multiple input single output (MISO) device.

9. The apparatus as described in claim 1 , wherein the hand gesture is a dynamic hand gesture and the processing element is configured to automatically recognize the hand gesture and to determine a command associated with the dynamic hand gesture.

10. The apparatus as described in claim 1 , wherein the plurality of receivers is spatially distributed.

11. The apparatus as described in claim 1 , wherein the transform over time process comprises a Fast Fourier Transform process.

12. The apparatus as described in claim 1 further comprising a microcontroller separate from the processing element and configured to perform static background subtraction.

13. The apparatus as described in claim 1 , wherein the determining a hand gesture based on the location and velocity data comprises a pattern recognition process by using a deep neural network.

14. The apparatus as described in claim 1 , wherein the transform over time process comprises a wavelet analysis process.

15. The apparatus as described in claim 1 , wherein the transform over time process comprises a super resolution process.

16. A system for hand gesture detection and classification, the system comprising:
a processing element;
a depth sensor communicatively coupled to the processing element;
a color sensor;
a transmitter configured for transmitting radar signals, wherein the transmitter is coupled to the processing element;
a plurality receivers comprising a first receiver, a second receiver, and a third receiver and configured for receiving the radar signals, and wherein the first receiver and the second receiver are configured for measuring a horizontal component of a movement and the second receiver and the third receiver are configured for measuring a vertical component of a movement, and wherein the plurality of receivers are communicatively coupled to the processing element; and
the processing element is configured to:
cross-calibrate the color sensor, the depth sensor, and the plurality of receivers;
register information received from the plurality of receivers with information received from the depth sensor based on a cross-calibration result;
generate location and velocity data based on reflections of the radar signals received by the plurality of receivers and based on information generated by the depth sensor and by performing a transform over time process; and
determine a hand gesture based on the location and velocity data;

wherein the transmitter and the plurality of receivers are configured for short range radar based detection, and
wherein the location and velocity data are generated further based on information generated by the color sensor.

17. The system as described in claim 16 , wherein the short range based detection of the hand gestures has a range of one meter.

18. The system as described in claim 16 , wherein the transmitter and the plurality of receivers are a portion of a user interface device of a vehicle.

19. The system as described in claim 16 , wherein the processing element is configured to determine a 3D position and radial velocity for a signal reflection received from the plurality of receivers.

20. The system as described in claim 19 , wherein the processing element is further configured to determine an elevation angle and an azimuth angle for the signal reflection received from the plurality of receivers.

21. The system as described in claim 16 , wherein the transmitter and the plurality of receivers are operated in a frequency-modulated continuous-wave (FMCW) mode.

22. The system as described in claim 16 , wherein the transform over time process comprises a Fast Fourier Transform process.

23. The system as described in claim 16 , wherein determining a hand gesture based on the location and velocity data comprises a pattern recognition process by using a deep neural network.

24. The system as described in claim 16 , wherein the transform over time process comprises a wavelet analysis process.

25. The system as described in claim 4 , wherein the transform over time process comprises a super resolution process.

26. A mobile apparatus comprising:
a processing element;
a depth sensor communicatively coupled to the processing element;
a color sensor;
a transmitter configured for transmitting radar signals, wherein the transmitter is coupled to the processing element;
a plurality of receivers comprising a first receiver, a second receiver, and a third receiver and configured for receiving the radar signals, and wherein the first receiver and the second receiver are configured for measuring a horizontal component of a movement and the second receiver and the third receiver are configured for measuring a vertical component of a movement, and wherein the a plurality of receivers are communicatively coupled to the processing element; and
the processing element is configured to:
cross-calibrat

Assignees

• Nvidia Corp

Inventors

• Molchanov Pavlo
• Gupta Shalini
• Kim Kihwan
• Pulli Kari

Classifications

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06F18/25

  Fusion techniques · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title