Interactive and shared surfaces

Patent metadata

Field	Value
Publication number	US-9560314-B2
Application number	US-201113160094-A
Country	US
Kind code	B2
Filing date	Jun 14, 2011
Priority date	Jun 14, 2011
Publication date	Jan 31, 2017
Grant date	Jan 31, 2017

Abstract

The interactive and shared surface technique described herein employs hardware that can project on any surface, capture color video of that surface, and get depth information of and above the surface while preventing visual feedback (also known as video feedback, video echo, or visual echo). The technique provides N-way sharing of a surface using video compositing. It also provides for automatic calibration of hardware components, including calibration of any projector, RGB camera, depth camera and any microphones employed by the technique. The technique provides object manipulation with physical, visual, audio, and hover gestures and interaction between digital objects displayed on the surface and physical objects placed on or above the surface. It can capture and scan the surface in a manner that captures or scans exactly what the user sees, which includes both local and remote objects, drawings, annotations, hands, and so forth.

First claim

What is claimed is:

1. A computer-implemented process for making any surface interactive and shared, comprising:
establishing a collaborative session between a local location and at least one remote location;
using a color camera, a depth camera and a projector at each location, calibrating a local horizontal arbitrary surface at the local location and at least one remote horizontal arbitrary surface at a remote location;
simultaneously capturing color video and depth images of the local horizontal arbitrary surface at the local location;
separating the foreground and the background of the color video of the local horizontal surface;
transmitting the foreground of the color video of the local horizontal surface to at least one selected remote location;
projecting the foreground video of the local horizontal surface onto the remote horizontal surface of the at least one selected remote location to superimpose the color video of the local horizontal surface onto the remote horizontal surface at the selected remote location.

2. The computer-implemented process of claim 1 , further comprising the actions of:
capturing a user interacting with the remote surface or objects on the remote surface at the selected remote location without a visual echo of the projected foreground video of the local surface;
separating the foreground and the background of the video of the remote surface at the selected remote location;
transmitting the foreground video of the selected remote surface to the local location;
projecting the foreground video of the selected remote surface onto the local surface at the local location to superimpose the remote surface, portions of a user and objects on or above the remote surface of the selected remote location onto the local surface at the local location.

3. The computer-implemented process of claim 2 , wherein the visual echo of the local surface is eliminated by turning off the projected video of the local surface when capturing video of the selected remote surface.

4. The computer-implemented process of claim 2 , further comprising capturing the local surface comprising the projected foreground video of the remote surface onto the local surface, objects projected on the surface from other sources, objects on or above the local surface, and any writing added by the user on the local surface.

5. The computer-implemented process of claim 2 , further comprising:
a user placing a physical document or object on the remote surface at the selected remote location;
a user activating a capture function projected onto the remote surface at the selected remote location; and
capturing the physical document or object on the remote surface in an electronic format which does not contain any projected video.

6. The computer-implemented process of claim 2 , further comprising:
displaying a projected document on to the remote surface at the selected remote location;
a user drawing on the projected document with real or digital ink;
the user activating a capture function via a gesture or voice;
capturing the projected document to create a new electronic document containing the original projected document with an image of the user's drawing superimposed on it.

7. The computer-implemented process of claim 1 , wherein separating the foreground and the background of the video of the local surface, further comprises:
comparing red, green and blue values for each pixel in a captured frame of the video of the local surface to a corresponding pixel captured when calibrating the local surface at the local location, and
if the difference in red or green or blue value is above a prescribed threshold, considering that pixel as foreground, and considering that pixel as part of a foreground image, and otherwise considering the pixel as background.

8. The computer-implemented process of claim 1 , wherein separating the foreground and the background of the video of the local surface, further comprises:
comparing red, green and blue values for each pixel in a captured frame of the video of the local surface to a corresponding pixel captured when calibrating the local surface at the local location, and
if the difference in red or green or blue value is below a prescribed threshold, assigning a low transparency value to that pixel and considering that pixel as foreground, and if the difference in red or green or blue value is above a prescribed threshold, assigning a high transparency value to that pixel and considering that pixel as background.

9. The computer-implemented process of claim 1 , further comprising:
capturing color video of an additional remote surface at an additional remote location;
separating the foreground and the background of the color video of the additional remote surface;
transmitting the foreground video of the additional remote surface to the selected remote location;
projecting the foreground video of the local surface and the foreground video of the additional remote surface onto the remote surface at the selected remote location to superimpose the foreground of the local surface and the foreground of the additional remote surface onto the remote surface at the selected remote location.

10. The computer-implemented process of claim 1 , wherein capturing color video of the local surface at the local location and the remote surface at the selected remote location, further comprises:
at the local location, capturing color video of the local surface with a RGB camera and capturing corresponding video of the local surface with a depth camera; and
at the selected remote location, capturing the remote surface with a RGB camera and a capturing corresponding video of the remote surface with a depth camera.

11. The computer-implemented process of claim 10 , further comprising using a depth camera to determine where a user or an object is located relative to a surface.

12. A computer-implemented process for making any surface interactive and shared, comprising:
establishing a collaborative session between a first location, a second location and a third location;
calibrating a first RGB camera, a first depth camera and a first projector at the first location, a second RGB camera, a second depth camera and a second projector at the second location, and a third RGB camera, a third depth camera and a third projector at the third location;
capturing color video of a first location horizontal surface with the first RGB camera and corresponding video with the first depth camera at the first location;
capturing color video of a second location horizontal surface with the second RGB camera and corresponding video with the second depth camera at the second location;
separating the foreground and the background of the color video of the first location horizontal surface;
separating the foreground and the background of the color video of the second location horizontal surface;
transmitting the foreground video of the first location horizontal surface and the foreground video of the second horizontal location surface to the third location;
projecting the foreground video of the first location horizontal surface and the foreground video of the second location horizontal surface with the third projector onto a third location horizontal surface to superimpose objects on the first location surface and objects on the second surface onto the third location surface at the third location.

13. The computer-implemented process of claim 12 , wherein a depth camera at a location is used to determine a user's hand relative to a corresponding surface at that location and wherein the user's hand location is used to manipulate a projected obj

Assignees

• Junuzovic Sasa
• Blank William Thomas
• Cleary Iii Bruce Arnold
• Gupta Anoop
• Wilson Andrew D
• Microsoft Technology Licensing Llc

Inventors

• Junuzovic Sasa
• Blank William Thomas
• Cleary Iii Bruce Arnold
• Gupta Anoop
• Wilson Andrew D

Classifications

• G06Q10/101

  Collaborative creation, e.g. joint development of products or services · CPC title

• G01B11/24

  for measuring contours or curvatures · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/0425

  using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386) · CPC title

• H04N7/142 Primary

  Constructional details of the terminal equipment, e.g. arrangements of the camera and the display · CPC title