Sign language inputs to a vehicle user interface

Patent metadata

Field	Value
Publication number	US-11009963-B2
Application number	US-201616302242-A
Country	US
Kind code	B2
Filing date	May 20, 2016
Priority date	May 20, 2016
Publication date	May 18, 2021
Grant date	May 18, 2021

Abstract

Sensors in a vehicle detect execution of a sign language symbol. The sign language symbol invokes a vehicle interface function. In some embodiments, the sign language symbol is an ASL sign. Sensors include three-dimensional optical, ultrasonic, or other sensors. The locations of targets (fingertips, knuckles, palm, etc.) are detected in the individual outputs of the sensors. The locations of targets determined from the outputs of multiple sensors may be associated with one another and filtered, such as using an RBMCDA algorithm and Kalman filtering. The system may be calibrated to detect each symbol by measuring the target locations with the user executing the each symbol.

First claim

The invention claimed is:

1. A method comprising:
capturing, by a vehicle controller of a vehicle, three-dimensional (3D) data of a user inside the vehicle using a plurality of sensors;
determining, based on the 3D data, that the user executed a sign language symbol;
receiving a plurality of outputs from the plurality of sensors;
identifying, based on the plurality of outputs, a plurality of target locations corresponding to a portion of a hand of the user;
for each portion of a plurality of portions of the hand of the user:
associating a target location to a group of target f the hand of the user for each sensor of the plurality of sensors; and
filtering the group of target locations for the each portion to obtain a filtered target location; and
identifying the sign language symbol according to the filtered target locations for the plurality of portions of the hand of the user; and

invoking a vehicle interface function corresponding to the sign language symbol, wherein the vehicle interface function is output of a textual or audible translation of the sign language symbol.

2. The method of claim 1 , wherein the vehicle interface function is a navigation function.

3. The method of claim 1 , wherein the vehicle interface function is a sound system control function.

4. The method of claim 1 , wherein the sensor is a three-dimensional optical sensor.

5. The method of claim 1 , wherein the sensor is one of a plurality of sensors, the plurality of sensors including a plurality of three-dimensional optical sensors.

6. The method of claim 1 , wherein the sensor is one of a plurality of sensors, the plurality of sensors including at least one of a three-dimensional optical sensor and an ultrasonic sensor.

7. The method of claim 1 , wherein associating a target location to the group of target locations for the each portion of the hand of the user for each sensor of the plurality of sensors comprises performing a Rao-Blackwellized Monte Carlo Data Association (RBMCDA) algorithm.

8. The method of claim 7 , wherein temporally filtering the group of target locations for the each portion to obtain the filtered target location comprises performing Kalman filtering.

9. A vehicle comprising:
a plurality of vehicle systems housed in the vehicle;
a sensor; and
a controller housed in the vehicle and operably coupled to the plurality of vehicle systems and the sensor, the controller programmed to:
capture three-dimensional (3D) data of a user inside the vehicle using the sensor;
determine, based on the 3D data, that the user executed a sign language symbol;
identify, based on an output of the sensor, a plurality of target locations corresponding to a portion of a hand of the user;
associate a target location to a group of target locations for a plurality of portions of the hand of the user;
filter the group of target locations for the plurality of portions to obtain a filtered target location;
identify the sign language symbol according to the filtered target locations for the plurality of portions of the hand of the user; and
invoke a vehicle interface function of one of the plurality of vehicle systems corresponding to the sign language symbol, wherein the vehicle interface function is output of a textual or audible translation of the sign language symbol.

10. The vehicle of claim 9 , wherein the vehicle interface function is a navigation function.

11. The vehicle of claim 9 , wherein the vehicle interface function is a sound system control function.

12. The vehicle of claim 9 , wherein the sensor is a three-dimensional optical sensor.

13. The vehicle of claim 9 , wherein the sensor is one of a plurality of sensors, the plurality of sensors including a plurality of three-dimensional optical sensors.

14. The vehicle of claim 9 , wherein the sensor is one of a plurality of sensors, the plurality of sensors including at least one of a three-dimensional optical sensor and an ultrasonic sensor.

15. The vehicle of claim 1 , wherein the controller is further programmed to associate a target location to the group of target locations for the each portion of the hand of the user for each sensor of the plurality of sensors by performing a Rao-Blackwellized Monte Carlo Data Association (RBMCDA) algorithm.

16. The vehicle of claim 15 , wherein the controller is further programmed to temporally filter the group of target locations for the each portion to obtain the filtered target location by performing Kalman filtering.

Assignees

• Ford Global Tech Llc

Inventors

• Groh Alexander
• Hakeem Mohannad
• Khalid Umer

Classifications

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06V20/59

  inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions · CPC title

• G10H1/46

  Volume control · CPC title

• G10H1/00

  Details of electrophonic musical instruments · CPC title

• G01C21/34

  Route searching; Route guidance · CPC title