Pointer projection for natural user input

Patent metadata

Field	Value
Publication number	US-9823764-B2
Application number	US-201414559870-A
Country	US
Kind code	B2
Filing date	Dec 3, 2014
Priority date	Dec 3, 2014
Publication date	Nov 21, 2017
Grant date	Nov 21, 2017

Abstract

A method to identify a targeted object based on eye tracking and gesture recognition. The method is enacted in a compute system controlled by a user and operatively coupled to a machine vision system. In this method, the compute system receives, from the machine vision system, video imaging a head and pointer of the user. Based on the video, the compute system computes a geometric line of sight of the user, which is partly occluded by the pointer. Then, with reference to position data for one or more objects, the compute system identifies the targeted object, situated along the geometric line of sight.

First claim

The invention claimed is:

1. In a compute system operatively coupled to a machine vision system and to a display, a method comprising:
receiving, from the machine vision system, prior and subsequent video imaging a head and pointer of a user;
computing, based on the prior video, a prior geometric line of sight of the user partly occluded by the pointer;
assigning spatial coordinates to one or more non-display objects situated along the prior geometric line of sight, the spatial coordinates defining a position of each of the non-display objects within a field of view of the machine vision system;
computing, based on the subsequent video, a subsequent geometric line of sight of the user partly occluded by the pointer;
identifying, by reference to the spatial coordinates of the one or more non-display objects, a targeted display object situated on the display and along the subsequent geometric line of sight;
identifying in the subsequent video movement of the pointer along the subsequent geometric line of sight;
and executing code for a process associated with the targeted display object in response to the movement.

2. The method of claim 1 , further comprising presenting the targeted display object on the display.

3. The method of claim 1 , wherein computing the subsequent geometric line of sight includes computing spatial coordinates of the head from the subsequent video.

4. The method of claim 1 , wherein computing the subsequent geometric line of sight includes computing spatial coordinates of an eye or pupil of the user from the subsequent video.

5. The method of claim 1 , wherein computing the subsequent geometric line of sight includes
computing spatial coordinates of the pointer from the subsequent video,
and wherein the subsequent geometric line of sight is a straight line passing through the pointer and through the eye of the user.

6. The method of claim 1 , wherein identifying the targeted display object includes determining whether the targeted display object is situated along the subsequent geometric line of sight.

7. The method of claim 1 , wherein the pointer includes a finger of the user.

8. The method of claim 1 , wherein the machine vision system includes a depth camera, and wherein each of the prior and subsequent video includes a series of time-resolved depth images from the depth camera.

9. The method of claim 1 , wherein the machine vision system includes a flat-image camera, and wherein each of the prior and subsequent video includes a series of time-resolved images from the flat-image camera.

10. The method of claim 1 , wherein depth and flat-image cameras of the machine vision system have parallel optical axes oriented in the same direction.

11. The method of claim 1 , further comprising representing the targeted display object visually as a selected object in response to a selection command spoken by the user.

12. The method of claim 1 , further comprising receiving user input defining the spatial coordinates.

13. The method of claim 1 , wherein the one or more non-display objects define a viewing area of the display.

14. The method of claim 1 , further comprising:
receiving, from the machine vision system, final video imaging the head and pointer of the user;
computing, based on the final video, a final line of sight of the user partly occluded by the pointer; and
forming on the display, along the final line of sight, an image to represent the targeted display object.

15. The method of claim 1 , wherein the code is further executable to select the targeted display object in response to a selection command from the user.

16. The method of claim 1 , wherein the machine vision system and the display are integrated in a head-mounted device worn by the user.

17. In a compute system operatively coupled to a machine vision system and to a display, a method comprising:
receiving, from the machine vision system, video imaging a head and hand of a user;
computing, based on the video, a geometric line of sight of the user partly occluded by the hand;
identifying, with reference to position data for one or more objects, a targeted display object situated on the display and along the geometric line of sight;
identifying, in the video, movement of the hand away from the user, along the geometric line of sight;
and shifting a depth coordinate of the display object in response to movement of the hand away from the user, along the geometric line of sight.

18. The method of claim 17 further comprising:
processing the video to identify closure of the hand, with the hand remaining in the geometric line of sight; and
reducing the size of the display object in response to closure of the hand, with the hand maintained in the geometric line of sight.

19. In a compute system operatively coupled to a machine vision system, a method comprising:
associating a process of the compute system to a non-display object arranged in a field of view of the machine-vision system, the process being one of a plurality of processes associated with a corresponding plurality of non-display objects in the field of view of the machine-vision system;
receiving, from the machine vision system, video imaging a head and pointer of a user;
computing, based on the video, a geometric line of sight of the user partly occluded by the pointer;
identifying the non-display object by reference to position data defining spatial coordinates of the plurality of non-display objects, the non-display object being situated along the geometric line of sight;
identifying in the video movement of the pointer along the geometric line of sight;
and executing the process associated with the targeted non-display object in response to the movement.

20. The method of claim 19 , wherein the machine vision system is integrated into a head-mounted display device worn by the user.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Schwesinger Mark
• Leyvand Tommer
• Stachniak Szymon

Classifications

• G06F3/012 Primary

  Head tracking input arrangements · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/0488

  using a touch-screen or digitiser, e.g. input of commands through traced gestures · CPC title

• G06F3/0412 Primary

  Digitisers structurally integrated in a display · CPC title