Method and system enabling natural user interface gestures with an electronic system

Patent metadata

Field	Value
Publication number	US-9600078-B2
Application number	US-201414476662-A
Country	US
Kind code	B2
Filing date	Sep 3, 2014
Priority date	Feb 3, 2012
Publication date	Mar 21, 2017
Grant date	Mar 21, 2017

Abstract

An electronic device coupleable to a display screen includes a camera system that acquires optical data of a user comfortably gesturing in a user-customizable interaction zone having a z 0 plane, while controlling operation of the device. Subtle gestures include hand movements commenced in a dynamically resizable and relocatable interaction zone. Preferably (x,y,z) locations in the interaction zone are mapped to two-dimensional display screen locations. Detected user hand movements can signal the device that an interaction is occurring in gesture mode. Device response includes presenting GUI on the display screen, creating user feedback including haptic feedback. User three-dimensional interaction can manipulate displayed virtual objects, including releasing such objects. User hand gesture trajectory clues enable the device to anticipate probable user intent and to appropriately update display screen renderings.

First claim

What is claimed is:

1. A method to enable a user-object to communicate using gestures made in (x,y,z) space with an electronic device coupleable to a display screen whereon imagery is displayable, the method including the following steps:
(a) providing said electronic device with an optical acquisition system operable to capture image data of said user-object within a three-dimensional hover zone;
(b) defining within said three-dimensional hover zone an interaction subzone including at least one z 0 plane disposed intermediate a plane of said display screen and a plane at a maximum z-distance beyond which user-object gestures need not be recognized by said electronic device;
(c) processing image data captured at step (a) representing an interaction of said user-object with at least a portion of said z 0 plane, defined in step (b), to detect and to produce three-dimensional positional information of said interaction;
(d) using said three-dimensional positional information produced at step (c) to determine at least one of (i) when in time, and (ii) where in (x,y,z) space said user-object interacted with said z 0 plane;
(e) following determination at step (d), identifying a gesture being made by said user-object; and
(f) in response to identification of a gesture at step (e), altering at least one operating characteristic of said electronic device.

2. The method of claim 1 , wherein said user-object includes at least one of (i) at least a portion of a user's arm, (ii) at least a portion of a user's hand, (iii) at least a portion of a user finger, (iv) at least a portion of a user's thumb, and (v) at least a portions of a user's body.

3. The method of claim 1 , wherein step (a) includes providing at least one optical acquisition system selected from a group consisting of:
(i) a time-of-flight (TOF) system;
(ii) a structured light system;
(iii) a single two-dimensional camera system; and
(iv) an at least two two-dimensional camera system.

4. The method of claim 1 , wherein said device includes a device selected from a group consisting of (i) a smart TV, (ii) a tablet computer, (iii) a smart phone, (iv) a laptop computer, (iv) a netbook computer, (v) a device to which said display screen is not integrally attached, and (vi) an appliance.

5. The method of claim 1 , wherein in response to detection of user-object interaction, step (b) includes carrying out at least one step selected from a group consisting of:
(i) customizing location of said interaction subzone and said z 0 plane within;
(ii) accommodating comfort of said user-object during interaction by dynamically altering location of said interaction subzone and said z 0 plane within;
(iii) altering size of said interaction subzone and said z 0 plane within; and
(iv) dynamically altering appearance of at least a region of imagery rendered on said display screen.

6. The method of claim 1 , wherein at step (f) said electronic device creates at least one type of user feedback selected from a group consisting of (i) visual feedback presented on said display screen, (ii) acoustic feedback, and (iii) haptic feedback activating a user-object wearable haptic feedback receiving device.

7. The method of claim 1 , wherein step (f) further includes at least one of (i) rendering on said display screen a GUI commensurate with said gesture identified at step (e), (ii) anticipatorily distinguishing a portion of said display screen believed to be affected by an immediately following gesture, (iii) dynamically altering size of an object rendered on said display screen, (iv) dynamically altering a viewable characteristic of an object rendered on said display screen, and (v) altering an operating mode of said device responsive to a location of said user-object detected at step (c).

8. The method of claim 1 , wherein step (f) causes said electronic device to create at least one type of visual feedback selected from a group consisting of (i) displaying a ribbon menu responsive to an identified gesture that is directed at a specific region of said display screen, (ii) displaying a virtual cursor keypad responsive to an identified gesture directed at a specific region of said display screen, and (iii) displaying a dynamically size changing icon responsive to an identified gesture directed at a specific region of said display screen.

9. The method of claim 1 , wherein a recognized gesture includes at least one of (i) virtually touching an object rendered on said display screen with said user-object, (ii) virtually touching an object rendered on said display screen with said user-object and altering at least one of object size and object orientation, (iii) a gesture interacting with a three-dimensional object, having at least three virtual points thereon, rendered on said display screen, each of said three virtual points mappable to one of three points on said user-object such that said user-object can virtually grasp and manipulate said object, and can then virtually decouple from and release said object.

10. The method of claim 1 , wherein:
step (a) provides said optical acquisition system with at least two spaced-apart two-dimensional cameras whose respective fields of view (FOVs) intersect to define said hover zone; said cameras acquiring optical image data from at least N two-dimensional data point of a gesture made by said user-object without said user-object having to touch said display screen to interact with said z 0 plane;
step (c) processes two-dimensional image data acquired by said two spaced-apart two-dimensional cameras to carry out at least one processing step selected from a group consisting of:
(I) representing an imaged portion of said user-object user using a set of landmark points sufficient to recognize a gesture made by said user-object, and discarding at least about 90% of said number N data points, and representing an imaged portion of said user object using a set of landmark points sufficient to recognize a gesture made by said user-object;
(II) in addition to (I) above, further extracting from said number N of data points a lesser number of three-dimensional landmarks, wherein magnitude N of said three-dimensional landmarks has a characteristic selected from a group consisting of (i) said magnitude N≦100, (ii) said magnitude N≦50, and (iii) said magnitude N≦12;
(III) in addition to (II) above, further processing said two-dimensional image data so as to extract from said number N of data points a lesser number of three-dimensional landmarks such that processing reduces acquired image information to at least one magnitude selected from a group consisting of (i-1) ≦1% N, (ii-2) processing reduces acquired information to ≦0.1% N, and (iii-3) processing reduces acquired information to ≦0.01% N; and
(IV) (i-a) said processing includes edge detection, (ii-b) said processing includes object modeling, (iii-c) said processing includes image rectification, (iv-d) said processing includes epipolar geometric analysis, and (v-e) said processing includes machine learning;

wherein said landmark has at least one characteristic selected from a group consisting of (i) said landmark is approximate centroid of said user's head, (ii) said landmark is approximate centroid of said user's hand, (iii), said landmark is approximate centroid of said user's finger, and (iv) said landmark is approximate location of said user's fingertip.

11. A system enabling a user-object to communicate using gestures made in (x,y,z) space with an electronic device coupleable to a display screen whereon imagery is displayable, the system including:
an electronic device with an optical acquisition system operable to capture image data of said user-object within a three-dime

Assignees

• Aquifi Inc

Inventors

• Rafii Abbas

Classifications

• G06T15/20

  Perspective computation · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06F3/042

  by opto-electronic means · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title