Augmented reality using projector-camera enabled devices

Patent metadata

Field	Value
Publication number	US-9064312-B2
Application number	US-201213625657-A
Country	US
Kind code	B2
Filing date	Sep 24, 2012
Priority date	Sep 23, 2011
Publication date	Jun 23, 2015
Grant date	Jun 23, 2015

Abstract

An augmented reality scene may be registered onto an arbitrary surface. A camera may capture an image of the arbitrary surface. The camera may analyze the surface geometry of the arbitrary surface. In some embodiments, a processing computing device may analyze data captured by the camera and an adjacent camera to reconstruct the surface geometry of the arbitrary surface. A scene may be registered to a three dimensional coordinate system corresponding to the arbitrary surface. A projector may project the scene onto the arbitrary surface according to the registration so that the scene may not display as being distorted.

First claim

What is claimed is:

1. A system for projecting an image of a scene onto an arbitrary surface that is non-planar in both horizontal and vertical directions, the system comprising:
a first un-calibrated camera coupled to a first projector, the first un-calibrated camera being disposed to capture at least a first portion of the image projected onto the arbitrary surface by the first projector, wherein the first portion of the image includes a first point;
an adjacent un-calibrated camera coupled to a second projector, wherein the adjacent un-calibrated camera is disposed to capture at least a second portion of the image projected onto the arbitrary surface by the second projector, wherein the second portion of the image includes the first point; and
a processing computing device coupled to the first camera configured to:
recognize an underlying surface geometry of the arbitrary surface by analyzing data in the at least first portion of the image captured by the first camera and the at least second portion of the image captured by the adjacent camera,
register an overlapping portion of the image of the scene, that is projected by the first projector and projected by the second projector, within a three dimensional coordinate system for display on the arbitrary surface, and
control the at least one of the first projector and the second projector to display the overlapping portion of the image of the registered scene onto the arbitrary surface based on an analysis of the data in the at least first portion of the image captured by the first camera and the at least second portion of the image captured by the adjacent camera.

2. The system of claim 1 , wherein the arbitrary surface is non-planar.

3. The system of claim 1 , wherein the camera is uncalibrated prior to capturing the image.

4. The system of claim 3 , wherein the projector is uncalibrated prior to projecting the image.

5. The system of claim 1 , wherein the camera captures a field of view larger than a field of view of the projector.

6. The system of claim 1 , wherein the camera is disposed to capture a portion of an image associated with an adjacent projector.

7. The system of claim 1 , wherein the camera is configured to wirelessly communicate with adjacent cameras or projectors.

8. The system of claim 1 , wherein the system comprises a plurality of projectors coupled to the camera.

9. The system of claim 8 , wherein the system comprises a plurality of cameras coupled to the plurality of projectors.

10. A method for projecting an image of a scene onto an arbitrary surface that is non-planar in both horizontal and vertical directions, the method comprising:
capturing, from at least one un-calibrated camera, a set of images of the arbitrary surface while patterns are projected on the arbitrary surface by at least two projectors, wherein the arbitrary surface is horizontally and vertically irregular, and wherein at least a first portion of a first image projected by a first projector of the at least two projectors overlaps with at least a second portion of a second image projected by a second projector of the at least two projectors;
recognizing a surface geometry of the arbitrary surface, based on the patterns projected on the arbitrary surface, using stereo reconstruction to determine three dimensional locations of data points within the set of images; and
registering the first portion of the first image and the second portion of the second image to project the image of the scene corresponding to the surface geometry of the arbitrary surface.

11. The method of claim 10 , wherein the step of capturing includes using two adjacent cameras to capture the set of images.

12. The method of claim 11 , comprising recovering correspondences between the two adjacent cameras using binary structured light patterns (SLP) from a plurality of projectors.

13. The method of claim 11 , comprising: analyzing the set of images captured by the two adjacent cameras; and reconstructing the recognized surface geometry into a three dimensional image.

14. The method of claim 10 , wherein the step of registration includes associating points in the scene to corresponding three dimensional coordinates along the surface geometry of the arbitrary surface.

15. The method of claim 10 , comprising projecting a first image in the set of images to overlap an adjacent projected image in the set of images.

16. A non-transitory computer readable storage medium comprising:
a computer readable code, which when executed by a processing computing device, causes the processing computing device to:
capture an image of an arbitrary surface that is non-planar in both horizontal and vertical directions, wherein at least one point within the captured image is: within a field of view of an un-calibrated camera, within a first image projected by a first projector and within a second image projected by a second projector, or within an image projected by a projector and within a first field of view of a first un-calibrated camera and within a second field of view of a second un-calibrated camera;
analyze a surface geometry of the arbitrary surface, from the image of the arbitrary surface, for surface height variations; and
register a first portion of the first image, that is projected by the first projector, with a second portion of the second image, that is projected by the second projector, corresponding to the surface height variations of the arbitrary surface such that first height variations in the first portion of the first image and second height variations in the second portion of the second image are indiscernible from the surface height variations on which the first portion and the second portion are protected.

17. The non-transitory computer readable storage medium of claim 16 , wherein the computer readable code when executed by a processing computing device, relays data among adjacent cameras capturing images of the arbitrary surface.

18. The non-transitory computer readable storage medium of claim 16 , wherein the scene is registered to correspond to a three dimensional coordinate system associated with the arbitrary surface.

19. The non-transitory computer readable storage medium of claim 16 , wherein the computer readable code is configured, when executed by a processing computing device, to reconstruct the captured image using a non-linear optimization on data points representing the captured image.

20. The non-transitory computer readable storage medium of claim 16 , wherein the computer readable code is configured, when executed by a processing computing device, to control a projector to display the scene as corresponding to three dimensional coordinates along the surface geometry of the arbitrary surface.

Assignees

• Univ California

Inventors

• Majumder Aditi
• Sajadi Behzad

Classifications

• H04N9/3185

  Geometric adjustment, e.g. keystone or convergence (optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14) · CPC title

• H04N9/3147

  Multi-projection systems (video walls G06F3/1446) · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T3/0068 Primary

  Physics · mapped topic

• G09F19/226

  External wall display means; Facade advertising means · CPC title