User interface apparatus and control method

Patent metadata

Field	Value
Publication number	US-10310675-B2
Application number	US-201514818770-A
Country	US
Kind code	B2
Filing date	Aug 5, 2015
Priority date	Aug 25, 2014
Publication date	Jun 4, 2019
Grant date	Jun 4, 2019

Abstract

A three-dimensional image of an operation surface and a region upward thereof is acquired, a hand region is extracted from the three-dimensional image, and the position of a fingertip is specified based on the extracted hand region. A touch on the operation surface is detected based on the operation surface included in the three-dimensional image and the specified position of the fingertip, and if a touch is detected, the direction of the fingertip is specified, and a position obtained by shifting the position of the fingertip by a predetermined amount in the direction opposite to the specified direction of the fingertip is determined as the touch position.

First claim

What is claimed is:

1. An apparatus for specifying an operation performed on an operation surface, the apparatus comprising:
an image sensor installed at a position upward from the operation surface and configured to sense a distance from respective positions of a plurality of positions of a target object placed on the operation surface;
a memory storing instructions; and
one or more processors which execute the instructions and cause the apparatus to function as units comprising:
(1) a first acquisition unit that acquires distances from the image sensor to respective positions of a plurality of positions of a target object placed on the operation surface;
(2) a first determination unit that determines that the target object performs a touch gesture against the operation surface in a case where the distance acquired by the first acquisition unit is less than or equal to a predetermined value;
(3) a first specification unit that specifies a tip of the target object that is determined to have performed the touch gesture in an image having the distances acquired by the first acquisition unit;
(4) a second specification unit that specifies a direction of the target object in the image having the distances;
(5) a second acquisition unit that acquires an angle formed by the direction of the target object and the operation surface using the image having the distances; and
(6) a second determination unit that determines, as the touch position on the operation surface, a position shifted by a predetermined amount in the specified direction of the target object from the specified tip of the target object,
wherein the predetermined amount is a first amount in a case where the acquired angle is a first angle, and
wherein the predetermined amount is a second amount that is smaller than the first amount in a case where the acquired angle is a second angle that is larger than the first angle.

2. The apparatus according to claim 1 , wherein the second determination unit determines the touch position by shifting the position of the tip of the target object in a region of the target object projected onto the operation surface by the predetermined amount on the operation surface in the direction opposite to the direction of the target object, the shifted direction being determined according to the direction of a three-dimensional image of the target object in a three-dimensional space whose bottom surface is the operation surface.

3. The apparatus according to claim 1 , wherein the second determination unit determines the touch position by (a) shifting the position of the tip of the target object by the predetermined amount in the direction opposite to the direction of a three-dimensional image of the target object in a three-dimensional space whose bottom surface is the operation surface and (b) projecting the shifted position onto the operation surface.

4. The apparatus according to claim 3 , wherein the one or more processors execute the instructions and further cause the apparatus to function as a unit that acquires a color image of the operation surface and a region in a three-dimensional space whose bottom surface is the operation surface,
wherein the first specification unit specifies the position of the tip of the target object on the operation surface based on a color of a hand region in the color image, and
wherein the second determination unit (a) obtains, as a corrected position of the tip, a position extended to the position of the tip on the operation surface that was specified by the first specification unit in the direction of the target object in the image having the distances that was specified by the second specification unit, (b) shifts the corrected position of the tip by the predetermined amount in the direction opposite to the direction of the target object, (c) projects the shifted position onto the operation surface, and (d) determines the projected position as the touch position.

5. The apparatus according to claim 1 , wherein the second specification unit determines the center of a palm or the back of a hand, and specifies a direction of a fingertip from the center of a palm or the back of a hand as the direction of the tip of the target object.

6. The apparatus according to claim 1 , wherein the second determination unit determines a center of gravity of pixels forming a finger included in a predetermined peripheral region of the tip of the target object as the touch position.

7. The apparatus according to claim 3 , wherein the target object is a hand,
wherein the one or more processors further cause the apparatus to function as:
(1) a detection unit that detects a touch on the operation surface by a finger of the hand based on the operation surface and the position of the tip of the finger included in the three dimensional image;
(2) a judgment unit that judges that the tip of the finger is at a predetermined distance or less from the operation surface in a case where the detection unit has not detected a touch on the operation surface; and
(3) a measurement unit that measures a finger length from a region of the target object in a case where the tip of the finger is at the predetermined distance or less, and
wherein the second determination unit (a) obtains, as a corrected position of the tip of the finger, a position obtained by shifting the position of the tip of the finger by the finger length measured by the measurement unit in the direction of the target object in the image having the distances that was specified by the second specification unit, (b) shifts the corrected position of the tip of the finger by the predetermined amount in the direction opposite to the direction of the target object, (c) projects the shifted position onto the operation surface, and (d) determines the projected position as the touch position.

8. The apparatus according to claim 7 , wherein the measurement unit (a) calculates a curvature of an outline based on an outline of the region of the target object, and (b) specifies a point at which the curvature is smaller than a predetermined value as the position of the tip of the finger, and
wherein the measurement unit (a) specifies a location at which the region of the target object enters the operation surface based on an image obtained by projecting the three-dimensional image onto the operation surface, (b) specifies, beginning from the specified location, a position at which the width of the region of the target object is smaller than a predetermined threshold value as a finger base position, and (c) measures the length from the finger base position to the position of the tip of the finger as the finger length.

9. The apparatus according to claim 4 , wherein in a case where a degree of flatness threshold value for a corresponding region of the operation surface that has been stored in advance is larger than a predetermined value, the second determination unit (a) obtains the corrected position of the tip of the finger, (b) shifts the corrected position of the tip of the finger by the predetermined amount in the direction opposite to the direction of the tip of the finger, projects the shifted position onto the operation surface, and (c) determines the projected position as the touch position, and
wherein in a case where the degree of flatness threshold value is not larger than the predetermined value, the second determination unit (a) shifts the position of the tip of the finger that was specified by the first specification unit by the predetermined amount in the direction opposite to the direction of the tip of the finger, (b) projects the shifted position onto the operation surface, and (c) determines the projected position as the touch position.

10. The appara

Assignees

• Canon Kk

Inventors

• Oshima Soshi

Classifications

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F3/04883

  for inputting data by handwriting, e.g. gesture or text · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F2203/04101

  2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup · CPC title

• G06F3/0426

  tracking fingers with respect to a virtual keyboard projected or printed on the surface (virtual keyboards on touch screens G06F3/04886) · CPC title