Recognizing in-air gestures of a control object to control a vehicular control system

Patent metadata

Field	Value
Publication number	US-9868449-B1
Application number	US-201514725510-A
Country	US
Kind code	B1
Filing date	May 29, 2015
Priority date	May 30, 2014
Publication date	Jan 16, 2018
Grant date	Jan 16, 2018

Abstract

The technology disclosed relates to an embeddable motion sensory control device that detects gestures in a three dimensional (3D) sensory space within a vehicle cabin, detecting a gesture in the 3D sensory space and interpreting the gesture as a command to a (sub) system of the vehicle under control, and issuing the command when appropriate.

First claim

What is claimed is:

1. A method of recognizing in-air gestures of a control object to control a vehicular control system, the method including:
determining observation information characterizing in-air gestural motion of a control object from at least one image captured at time t 0 ;
constructing a three-dimensional model to represent the control object by fitting one or more three-dimensional solid subcomponents modelled in software to a construct of the control object defined by the observation information based on the image captured at time t 0 ;
responsive to modifications in the observation information based on another image captured at time t 1 , wherein the control object moved between t 0 and t 1 , improving conformance of the three-dimensional model to the modified observation information by:
determining an error indication including a difference in space between a point on another construct of the control object defined by the observation information based on the image captured at time t 1 and a corresponding point on at least one of the three-dimensional solid subcomponents modelled in software fitted to the construct defined by the observation information based on the image captured at time t 0 ; and responsive to the error indication adjusting the three-dimensional model; and

interpreting motions of the control object wherein the motions are determined based at least in part from the adjusting the three-dimensional model as one or more gestures providing command input to a vehicular control system; wherein the at least one image captured at time t 0 and the another image captured at time t 1 include:
capturing with a camera an ambient capture for a field of view of the camera in ambient lighting conditions;
illuminating a flash;
capturing with the camera a flash capture for the field of view of the camera while illuminated by the flash;
differencing the ambient capture and the flash capture to produce a clean image capture with reduced ambient lighting effects;
analyzing the clean image capture to construct a three-dimensional model of a control object in the field of view;
tracking, using the three-dimensional model, the control object by repeatedly producing and analyzing clean image captures of the control object and interpreting the gestures and outputting one or more control streams including signal which results in control of a vehicle system based on the interpretation of the gestures; and
wherein one or more sensors alternate between ambient captures and flash captures during at least part of gesture capture and analysis.

2. The method of claim 1 , the method further including:
scanning the field of view of the camera by selectively illuminating for short durations respective ones of a pre-determined plurality of directionally oriented light sources that have overlapping fields of illumination;
measuring one or more differences in intensity of returning light emitted from the respective light sources and reflected from a control object in a vehicle environment;
analyzing the reflected light across a series of image frames for periodic brightness variations; and
determining positional information of the control object based at least in part upon one or more measured differences in intensity of the returning light.

3. The method of claim 2 , wherein the short durations are in order of 5 to 40 microseconds.

4. The method of claim 2 , wherein the pre-determined plurality of directionally oriented light sources includes between six to sixteen light sources.

5. The method of claim 2 , wherein a rate of selective illumination of the light sources is determined by measuring one or more frequencies of the periodic brightness variations.

6. The method of claim 2 , further including varying a rate of capture of the series of image frames.

7. The method of claim 2 , wherein selectively illuminating the respective light sources further includes varying brightness of pairs of overlapping light sources by dimming a first, initially on light source while brightening a second, initially off light source.

8. The method of claim 2 , further including capturing the image frames at a low-resolution by analyzing a reduced amount of image data lines per image frame.

9. The method of claim 1 , wherein the flash has a duration of 5 to 300 microseconds.

10. The method of claim 1 , wherein the flash has a duration of 5 to 40 microseconds.

11. The method of claim 1 , wherein the flash is produced by six to sixteen light emitting diodes.

12. The method of claim 1 , wherein the flash is produced by light emitting diodes designed to mix wide beams and narrow beams.

13. The method of claim 1 , wherein the flash is produced by light emitting diodes operating in an ultraviolet frequency range and the capturing further includes using an ultraviolet (UV) sensitive or UV filtered camera.

14. The method of claim 1 , wherein the flash is produced by light emitting diodes operating in an infrared (IR) frequency range and the capturing further includes using an IR sensitive or IR filtered camera.

15. The method of claim 1 , wherein the flash capture is timed with the flash to reduce capture from ambient lighting.

16. The method of claim 1 , wherein duration of the flash capture lasts no more than four times the duration of the flash.

17. The method of claim 1 , wherein the one or more sensors operate, at least part time, at a reduced resolution that is half or less of available sensor resolution and at an increased capture rate enabled by the reduced resolution.

18. The method of claim 1 , wherein one or more sensors capture binocular views in of at least the flash captures.

19. The method of claim 18 , further including producing clean binocular captures by differencing and constructing the three-dimensional model of the control object from clean binocular views.

20. The method of claim 19 , wherein the differencing to produce clean captures further includes comparing the flash capture to one or more ambient captures immediately before or after the flash capture.

21. The method of claim 19 , wherein the differencing to produce clean captures further includes comparing the flashes capture to two or more ambient captures immediately before and after the flash capture.

22. The method of claim 1 , further including:
capturing at least flash images of a passenger vehicle compartment and creating a 3D model of non-control object features; and
when constructing and following gestures of the 3D model of the control object, taking into account the non-control object features of the passenger vehicle compartment.

Assignees

• Leap Motion Inc

Inventors

• Holz David S
• Yang Hua
• Gordon Robert Samuel
• Roy Neeloy
• Schunick Justin
• Durdik Paul A

Classifications

• G06K9/00362

  Physics · mapped topic

• G06T2207/10152

  Varying illumination · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• B60W2540/00

  Input parameters relating to occupants · CPC title

• B60W50/10 Primary

  Interpretation of driver requests or demands · CPC title