Obtaining input from a virtual user interface

Patent metadata

Field	Value
Publication number	US-9703371-B1
Application number	US-201514753745-A
Country	US
Kind code	B1
Filing date	Jun 29, 2015
Priority date	Mar 11, 2013
Publication date	Jul 11, 2017
Grant date	Jul 11, 2017

Abstract

A virtual user interface in a 3D environment is projected onto a display object, such as an open hand. An operating object, such as a stylus or a second hand with an index finger extended, may be used to select input features of the projected user interface. In one scenario, for example, a telephone keypad may be projected onto the open hand and the index finger may be used to select a particular number on the keypad. Images of the hands may be segmented to produce contours of the finger and open hand, and these contours are used to determine the location of the finger relative to the open hand.

First claim

What is claimed is:

1. A method comprising:
determining first curvature information associated with a first object;
determining second curvature information associated with a second object;
projecting at least a portion of an image onto the first object;
detecting that a portion of the second object overlaps at least a portion of the first object; and
determining a location of a selected image portion of the image based at least in part on the first curvature information and the second curvature information.

2. The method of claim 1 , wherein the second object is a hand.

3. The method of claim 1 , further comprising:
forming a merged contour from the first curvature information and the second curvature information; and
wherein determining that the second object overlaps the first object is based at least in part on the merged contour.

4. The method of claim 1 , further comprising determining the second object is within a distance threshold of the first object prior to determining the location.

5. The method of claim 1 , further comprising applying a Gaussian filter to generate at least one of the first contour information or the second contour information.

6. The method of claim 1 , further comprising:
determining a length of the portion of the second object that overlaps with the portion of the first object; and
wherein determining the location is based at least in part on the length.

7. A method comprising:
under control of one or more computer systems configured with executable instructions,
generating first curvature information related to a display object and second curvature information related to an operating object within a first image projected into an environment;
determining one or more continuous surfaces in a second image projected into the environment;
generating contours of surfaces in the second image, the surfaces including at least a portion of the display object and at least a portion of the operating object; and
determining a location based at least in part on the contours.

8. The method of claim 7 , wherein determining the location is based at least in part on the first curvature information and the second curvature information.

9. The method of claim 7 , further comprising:
detecting a first contour within the first image, the contour associated with the display object;
detecting a second contour within the first image, the contour associated with the display object;
wherein the first curvature information is determined based at least in part on the first contour; and
wherein the second curvature information is determined based at least in part on the second contour.

10. The method of claim 7 , further comprising determining at least one boundary associated with the display object and the operating object and wherein determining the location is based at least in part on the at least one boundary.

11. The method of claim 7 , further comprising performing at least one operation, the at least one operation associated with the location on the display object.

12. The method of claim 7 , wherein the second image is a graphical user interface.

13. The method of claim 7 , further comprising:
segmenting the first image to produce contours prior to generating the first curvature information and the second curvature information.

14. A system comprising:
one or more processors;
a component for receiving image data;
one or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising:
generating curvature information of a display object and an operating object based at least in part on image data received from the component, the display object and the operating object associated with a projection area; and
determining a location associated with the operating object based at least in part on the curvature information of the display object and the operating object.

15. The system of claim 14 , wherein the component for receiving the image data includes a camera.

16. The system of claim 14 , further comprising:
a projector for projecting an image into the projection area; and
wherein the image data is received by the component while the image is projected into the projection area.

17. The system of claim 14 , wherein the one or more computer-readable media stores computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising determining that the operating object overlaps the display object.

18. The system of claim 14 , wherein the one or more computer-readable media stores computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising:
determining contour information associated with the display object and the operating object based on the image data received from the component; and
wherein the curvature information is generated based at least in part on the contour information.

19. The system of claim 14 , wherein the one or more computer-readable media stores computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising determining a distance between the operating object and at least one other object within the projection area.

20. The system of claim 14 , wherein generating the curvature information of objects associated with the projection area includes
segmenting the image to produce contours.

Assignees

• Amazon Tech Inc

Inventors

• Yao Ning
• Chang Samuel Henry

Classifications

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title