Dynamic selection of surfaces in real world for projection of information thereon

Patent metadata

Field	Value
Publication number	US-9245193-B2
Application number	US-201213549388-A
Country	US
Kind code	B2
Filing date	Jul 13, 2012
Priority date	Aug 19, 2011
Publication date	Jan 26, 2016
Grant date	Jan 26, 2016

Abstract

One or more devices capture a scene of real world, and process one or more image(s) which include distances to points on surfaces in the real world. The distances are used to automatically identify a set of surfaces in the real world. Then, the one or more devices check whether a surface in the set is suitable for display of an element of information to be projected into the scene. On finding that a surface is suitable, a transform function is automatically identified, followed by automatic application of the transform function to the element of the information. A transformed element, which results from automatically applying the transform function, is stored in a frame buffer coupled to a projector, at a specific position in the frame buffer identified during the check for suitability. When no surface is suitable, user input is obtained, followed by projection of information as per user input.

First claim

What is claimed is:

1. A method comprising:
receiving an image of a scene in real world, the image comprising a plurality of distances to points in the real world;
using at least the plurality of distances, to identify a set of surfaces in the scene;
selecting an element of information to project on to the scene, the selected element of information having a size, the information comprising multiple elements that are independent of one another and can be projected on different surfaces;
comparing the size of the selected element to a size of a surface in the set of surfaces;
selecting the surface from the set of surfaces, on which the selected element of information can be projected, selection of the surface being based at least in part on the size of the surface being found by the comparing to be sufficiently large to accommodate the element;
one or more processors identifying a transform function to use the surface;
applying to at least the selected element of information, the transform function; and
storing in a frame buffer in a memory, at a specific position, a transformed element resulting from the applying.

2. The method of claim 1 wherein:
an area of a rectangle to display the selected element of information is compared to a total number of pixels in a region of the image corresponding to the surface.

3. The method of claim 1 wherein:
the selection of the surface comprises testing whether all pixels in a rectangle to be used to project the selected element of information are present in a region of the image corresponding to the surface.

4. The method of claim 1 wherein:
during selection of the surface, another specific position is identified on the surface, and stored in the memory.

5. The method of claim 3 wherein:
the testing is performed repeatedly, to associate with the selected element of information, multiple surfaces that form a subset of the set of surfaces, such that the selected element of information can be projected on each of the multiple surfaces;
the method further comprises assigning projection of the selected element of information to the surface which is selected from the subset.

6. The method of claim 1 further comprising:
operating a projector to project the transformed element in the frame buffer;
receiving a user input, subsequent to the operating; and
in response to receipt of the user input, repeating for a different surface, the identifying of the transform function, the applying and the storing.

7. The method of claim 6 wherein:
the user input identifies the different surface.

8. The method of claim 6 wherein:
the user input is identified as a hand gesture in another image.

9. The method of claim 1 further comprising:
receiving user input identifying the different surface when the finding indicates the selected element of information cannot be projected on any surface in the set of surfaces; and
repeating for the different surface, the identifying of the transform function, the applying and the storing.

10. The method of claim 1 further comprising:
receiving user input identifying a shape; and
using the shape in the transform function, to mask out a portion of the selected element of information.

11. One or more non-transitory computer readable storage media comprising:
instructions to receive an image of a scene in real world, the image comprising a plurality of distances to points in the scene;
instructions to use at least the plurality of distances in the image, to identify a set of surfaces in the scene;
instructions to select an element of information to project on to the scene, the selected element of information having a size, the information comprising multiple elements that are independent of one another and can be projected on different surfaces;
instructions to compare the size of the selected element to a size of a surface in the set of surfaces;
instructions to select the surface from the set of surfaces, on which the selected element of information can be projected, selection of the surface being based at least in part on the size of the surface being found by execution of the instructions to compare to be sufficiently large to accommodate the element;
instructions to one or more processors to identify a transform function to use the surface
instructions to apply to at least the selected element of information, the transform function; and
instructions to store in a frame buffer in a memory, at a specific position, a transformed element resulting from execution of the instructions to apply.

12. The one or more non-transitory computer readable storage media of claim 11 wherein:
the instructions to select comprise instructions to compare a total number of pixels in a region of the image corresponding to the surface with an area of a rectangle to display the selected element of information.

13. The one or more non-transitory computer readable storage media of claim 11 wherein:
the instructions to select comprise instructions to test whether all pixels in a rectangle to be used to display the selected element of information are present in a region of the image corresponding to the surface.

14. The one or more non-transitory computer readable storage media of claim 13 wherein:
another specific position is identified on the surface by execution of the instructions to select.

15. The one or more non-transitory computer readable storage media of claim 13 wherein:
the instructions to test are configured to be executed repeatedly, to associate with the selected element of information, multiple surfaces that form a subset of the set of surfaces, such that the selected element of information can be projected on each of the multiple surfaces;
the one or more non-transitory computer readable storage media further comprise instructions to assign projection of the selected element of information to the surface which is selected from the subset.

16. The one or more non-transitory computer readable storage media of claim 11 wherein:
instructions to operate a projector to project the transformed element in the frame buffer; and
instructions to receive a user input, subsequent to the operating; and
instructions to execute for a different surface, the instructions to identify the transform function, the instructions to apply and the instructions to store.

17. The one or more non-transitory computer readable storage media of claim 16 wherein:
the user input identifies the different surface.

18. The one or more non-transitory computer readable storage media of claim 16 wherein:
the user input is identified in at least another image.

19. The one or more non-transitory computer readable storage media of claim 11 further comprising:
instructions to receive a user input identifying the different surface, configured to be executed when execution of the instructions to check indicates the selected element of information cannot be projected on any surface in the set of surfaces; and
instructions to repeat for the different surface, the instructions to identify the transform function, the instructions to apply and the instructions to store.

20. The one or more non-transitory computer readable storage media of claim 11 further comprising:
instructions to receive a user input identifying a shape; and
instructions to use the shape in the transform function, to mask out a portion of the selected element of information.

21. One or more devices comprising:
a camera;
one or more processors

Assignees

• Kulkarni Tejas Dattatraya
• Qualcomm Inc

Inventors

• Kulkarni Tejas Dattatraya

Classifications

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• H04N9/3194

  including sensor feedback · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06K9/228 Primary

  Physics · mapped topic