Bimanual interactions on digital paper using a pen and a spatially-aware mobile projector

Patent metadata

Field	Value
Publication number	US-9513716-B2
Application number	US-201113040238-A
Country	US
Kind code	B2
Filing date	Mar 3, 2011
Priority date	Mar 4, 2010
Publication date	Dec 6, 2016
Grant date	Dec 6, 2016

Abstract

One embodiment of the present invention sets forth a technique for providing an end user with a spatially-aware projector combined with a spatially-aware digital pen for use in a design environment. Paper documents are augmented to allow a user to access additional information and computational tools through projected interfaces. Virtual ink may be managed in single and multi-user environments to enhance collaboration and data management. The spatially-aware projector pen provides end-users with dynamic visual feedback and improved interaction capabilities.

First claim

We claim:

1. A spatially-aware projector system, comprising:
a first position tracking mechanism configured to provide first position data related to a current position of a projector within a display surface;
a second position tracking mechanism configured to provide second position data related to a current position of a spatially-aware digital pen within the display surface; and
the projector configured to:
display a projected image generated from design data retrieved from a memory based on the first position data;
receive drawing input via the second position tracking mechanism;
analyze the drawing input relative to the projected image;
update the design data based on the drawing input; and
display a radial user interface menu including a plurality of regions separated by boundaries, wherein a first region in the plurality of regions is selected based on the spatially-aware digital pen remaining fixed and in contact with the display surface while the projector is relocated.

2. The system of claim 1 , further comprising a processor configured to or configurable to generate the projected image.

3. The system of claim 2 , further comprising an enclosure configured to house the first position tracking mechanism and the projection unit.

4. The system of claim 3 , wherein the enclosure further houses the processor.

5. The system of claim 3 , wherein the enclosure is further configured to be grasped by a hand of a user such that the user can position the spatially-aware projector within the display surface.

6. The system of claim 5 , further comprising an adjustable mirror for controlling the size of the projected image across the display surface.

7. The system of claim 6 , further comprising a swivel that may be manipulated by a user to adjust the mirror to change the size of the projected image within the display surface.

8. The system of claim 5 , further comprising a scroll-wheel that, when rotated, causes the size of at least one element in the projected image to change.

9. The system of claim 1 , wherein the spatially-aware digital pen further includes a force sensor that detects when the spatially-aware digital pen contacts the display surface.

10. The system of claim 9 , further comprising a processor configured to update the projected image to include the radial user interface menu when the spatially-aware digital pen is contacted to and held in place against the display surface for a threshold amount of time.

11. The system of claim 10 , wherein each of the regions included in the plurality of regions represents a different menu item included in the radial menu.

12. The system of claim 1 , wherein selecting the first region in the plurality of regions is further based on a virtual cursor that is controlled by movements of the spatially-aware digital pen crossing a boundary delineating the first region while the spatially-aware digital pen remains fixed and in contact with the display surface.

13. The system of claim 9 , wherein virtual ink strokes are collected and stored as image data in memory when the spatially-aware digital pen is contacted to and moved across the display surface.

14. The system of claim 13 , further comprising a processor configured to update the projected image to include the image data.

15. The system of claim 1 , wherein the spatially-aware digital pen selects an element that is outside of the boundaries of the projected image to produce a copy of the element.

16. The system of claim 15 , further comprising a processor further configured to receive a paste position on the display surface and within the boundaries of the projected image and update the projected image to display the copy of the element at the paste position within the projected image.

17. The system of claim 5 , wherein the processor is further configured to or configurable to:
update the projected image to display a plurality of user interface elements, wherein each user interface element is partially transparent and includes a border of a different color;
determine an element selected via the spatially-aware digital pen lies within the boundaries of a first user interface element included in the plurality of user interface elements; and
update the selected element to match the border color of the first user interface element.

18. The system of claim 1 , further comprising a processor configured to render a sequence of frames of pixel data based on the design data and a drawing path captured by the spatially-aware digital pen to produce a walk-through animation of a three-dimensional model represented by the design data.

19. The system of claim 5 , further comprising a network interface that is coupled to the processor and configured to interface with an additional spatially-aware digital pen operated by another user.

20. The system of claim 1 , further comprising a mechanical controller that, when adjusted, changes a spread between each successive raster line produced by the projection unit.

21. A spatially-aware projector system, comprising:
a projector that includes a first position tracking mechanism configured to provide position data related to a current position of the projector within a display surface, the projector configured to display a projected image generated from design data retrieved from a memory based on the position data;
a spatially-aware digital pen that includes a second position tracking mechanism configured to provide position data related to a current position of the spatially-aware digital pen within the display surface; and
a processor coupled to the spatially-aware digital pen and configured to:
receive drawing input corresponding to a drawing path via the second position tracking mechanism;
analyze the drawing input relative to the projected image; and
render a sequence of frames of pixel data based on the design data and the drawing path to produce a walk-through animation of a three-dimensional model represented by the design data.

22. A spatially-aware projector system, comprising:
a first position tracking mechanism configured to provide position data related to a current position of the spatially-aware projector within a display surface;
a projection unit configured to display a projected image generated from design data retrieved from memory based on the position data;
a processor configured to or configurable to:
generate the projected image, and
update the projected image to include a radial user interface menu when the spatially-aware digital pen is contacted to and held in place against the display surface for a threshold amount of time,
wherein the radial user interface includes a plurality of regions separated by boundaries such that each of the regions represents a different menu item included in the radial menu, and

wherein a first region in the plurality of regions is selected when the spatially-aware projector is relocated and the spatially aware digital pen remains fixed and in contact with the display surface causing a boundary delineating the first region to be crossed by a virtual cursor that is controlled by movements of the spatially-aware digital pen;

an enclosure configured to:
house the first position tracking mechanism and the projection unit, and be grasped by a hand of a user such that the user can position the spatially-aware

digital pen that includes:
a second position tracking mechanism configured to provide position data related to a current position of the spatially-aware digital pen

Assignees

• Song Hyunyoung
• Guimbretiere Francois V
• Grossman Tovi
• Fitzmaurice George
• Autodesk Inc

Inventors

• Song Hyunyoung
• Guimbretiere Francois V
• Grossman Tovi
• Fitzmaurice George

Classifications

• G06F3/033 Primary

  Pointing devices displaced or positioned by the user {, e.g. mice, trackballs, pens or joysticks}; Accessories therefor (digitisers characterised by the transducing means G06F3/041) · CPC title

• H04N9/3185

  Geometric adjustment, e.g. keystone or convergence (optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14) · CPC title

• H04N9/3194

  including sensor feedback · CPC title

• G06F3/0425

  using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386) · CPC title

• G06F3/03545

  Pens or stylus · CPC title