Augmented reality surface segmentation

Patent metadata

Field	Value
Publication number	US-9530232-B2
Application number	US-201313802194-A
Country	US
Kind code	B2
Filing date	Mar 13, 2013
Priority date	Sep 4, 2012
Publication date	Dec 27, 2016
Grant date	Dec 27, 2016

Abstract

Methods, systems, computer-readable media, and apparatuses for providing intuitive, functional, and convenient ways of enabling a user of a head-mounted display unit or another augmented reality enabled device to interact with various user interfaces and other features provided by such a unit or device are presented. In some embodiments, a computing device, such as a head-mounted display unit, may receive camera input of a scene. Subsequently, the computing device may identify at least one reference object in the scene, for example, based on detecting one or more rectangles in the received camera input. The computing device then may receive input that defines a surface segment relative to the at least one reference object. Thereafter, the computing device may render the surface segment.

First claim

What is claimed is:

1. A method comprising:
receiving a first camera input of a scene;
identifying at least one reference object in the scene, wherein the at least one reference object is a physical object;
receiving a second camera input comprising user input from a first user defining a surface segment relative to the at least one reference object, wherein the user input defining the surface segment is a finger movement defining a shape of the surface segment; and
causing the surface segment to be rendered in the first user's field of vision as though on a surface on which the physical object is located.

2. The method of claim 1 , wherein identifying at least one reference object in the scene comprises detecting one or more feature points in the first camera input.

3. The method of claim 1 , wherein identifying at least one reference object in the scene comprises receiving a wireless communication from the at least one reference object.

4. The method of claim 1 , wherein identifying at least one reference object in the scene comprises detecting one or more rectangles in the first camera input.

5. The method of claim 4 , further comprising:
determining, based on the one or more detected rectangles, a current perspective of the scene, wherein the surface segment is rendered relative to the current perspective of the scene.

6. The method of claim 5 , further comprising:
determining that the current perspective of the scene has changed; and
dynamically updating the rendered surface segment.

7. The method of claim 1 , wherein the finger movement is performed by the first user of a device causing the surface segment to be rendered.

8. The method of claim 7 , wherein causing the surface segment to be rendered includes:
rendering, in a virtual workspace, the surface segment at a first angle; and
rendering, in the virtual workspace, a second surface segment defined by the first user at a second angle different from the first angle.

9. The method of claim 8 , further comprising:
determining that the at least one reference object has been removed from the scene; and
closing the virtual workspace.

10. The method of claim 9 , further comprising:
determining that the at least one reference object has been reintroduced into the scene or another scene; and
opening the virtual workspace.

11. The method of claim 8 , further comprising:
detecting a second reference object different from the at least one reference object; and
opening a second virtual workspace different from the virtual workspace.

12. The method of claim 1 , wherein the identifying the at least one reference object is based on an image displayed by the at least one reference object.

13. The method of claim 1 , further comprising:
transmitting a data signal comprising a virtual workspace that includes the surface segment to an at least one other device,
wherein the virtual workspace is defined by the first user that provided the input defining the surface segment, and
wherein the at least one other device is associated with a second user different from the first user.

14. The method of claim 13 ,
wherein causing the surface segment to be rendered comprises:
rendering the virtual workspace for the first user; and
based on interaction of the second user with one or more virtual objects included in the virtual workspace, dynamically updating the rendering of the virtual workspace for the first user.

15. The method of claim 1 , further comprising:
determining a pose of the at least one reference object based at least in part on the camera input; and
determining a virtual surface separate from the at least one reference object based at least in part on the determined pose, wherein the surface segment is rendered on the virtual surface.

16. The method of claim 1 , wherein the first camera input and the second camera input are different signals.

17. A system comprising:
an optical sensor configured to transmit a first input signal comprising an image of a scene;
a display;
a processor configured to:
identify at least one reference object in the scene, wherein the at least one reference object is a physical object;
receive a second input signal from the optical sensor, the second input signal comprising user input from a first user defining a surface segment relative to the at least one reference object, wherein the user input defining the surface segment is a finger movement defining a shape of the surface segment; and
render the surface segment in the display to appear in the first user's field of vision as though on a surface on which the physical object is located.

18. The system of claim 17 , wherein the processor is configured to identify the at least one reference object in the scene based on detecting one or more feature points in the image of the scene.

19. The system of claim 17 , wherein the processor is configured to identify the at least one reference object in the scene based on receiving a wireless communication from the at least one reference object.

20. The system of claim 17 , wherein the processor is configured to identify the at least one reference object in the scene based on detecting one or more rectangles in the image of the scene.

21. The system of claim 20 , wherein the processor is further configured to:
determine, based on the one or more detected rectangles, a current perspective of the scene,
wherein the surface segment is rendered relative to the current perspective of the scene.

22. The system of claim 21 , wherein the processor is further configured to:
determine that the current perspective of the scene has changed; and
dynamically update the rendered surface segment.

23. The system of claim 17 , wherein the finger movement is performed by the first user of a device causing the surface segment to be rendered.

24. The system of claim 23 , wherein rendering the surface segment includes:
rendering, in a virtual workspace, the surface segment at a first angle; and
rendering, in the virtual workspace, a second surface segment defined by the first user at a second angle different from the first angle.

25. The system of claim 24 , wherein the processor is further configured to:
determine when the at least one reference object has been removed from the scene; and
close the virtual workspace in response to determining that the at least one reference object has been removed.

26. The system of claim 25 , wherein the processor is further configured to:
determine that the at least one reference object has been reintroduced into the scene or another scene; and
open the virtual workspace in response to determining the at least one reference object has been reintroduced into the scene or another scene.

27. The system of claim 24 , wherein the processor is further configured to:
detect a second reference object different from the at least one reference object; and
open a second virtual workspace different from the virtual workspace.

28. The system of claim 17 , wherein the identifying the at least one reference object is based on an image displayed by the at least one reference object.

29. The system of claim 17 , further comprising:
a transmitter configured to transmit a data signal comprising a virtual workspace that includes the surface segment to at least one other device,
wherein the virtual wor

Assignees

• Qualcomm Inc

Inventors

• Maciocci Giuliano
• Mulholland James Joseph

Classifications

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06K9/00671

  Physics · mapped topic

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06K9/00355

  Physics · mapped topic