System and method for automatic alignment and projection mapping

Patent metadata

Field	Value
Publication number	US-10089778-B2
Application number	US-201514820693-A
Country	US
Kind code	B2
Filing date	Aug 7, 2015
Priority date	Aug 7, 2015
Publication date	Oct 2, 2018
Grant date	Oct 2, 2018

Abstract

A projector and at least two cameras are mounted with fields of view that overlap a projection area on a three-dimensional environment. A computing device: controls the projector to project structured light patterns that uniquely illuminate portions of the environment; acquires images of the patterns from the cameras; generates a two-dimensional mapping of the portions between projector and camera space and by processing the images and correlated patterns; generates a cloud of points representing the environment using the mapping and camera positions; determines a projector location, orientation and lens characteristics from the cloud; positions a virtual camera relative to a virtual three-dimensional environment, corresponding to the environment, parameters of the virtual camera respectively matching parameters of the projector; and, controls the projector to project based on a virtual location, orientation and characteristics of the virtual camera.

First claim

What is claimed is:

1. A system comprising:
a computing device; a projector; at least two cameras, each of the projector and the at least two cameras configured for mounting relative to a three-dimensional environment with respective fields of view at least partially overlapping a projection area of the projector on the three-dimensional environment, the three-dimensional environment comprising a physical object; and a memory storing a virtual model of the physical object; the computing device configured to:
control the projector to sequentially project one or more structured light patterns configured to uniquely illuminate different portions of the three-dimensional environment and the physical object;
acquire one or more respective images from each of the at least two cameras while the projector is projecting the one or more structured light patterns, each of the one or more respective images correlated with a given respective structured light pattern;
generate a two-dimensional mapping of the different portions of the three-dimensional environment between a projector space and a camera space by processing the respective images and correlated given respective structured light patterns;
generate a cloud of points representing the three-dimensional environment using the two-dimensional mapping and given positions of the at least two cameras relative to the three-dimensional environment;
determine a location, an orientation and lens characteristics of the projector relative to the three-dimensional environment, from the cloud of points;
position a virtual camera relative to a virtual three-dimensional environment, corresponding to the three-dimensional environment, a virtual location, a virtual orientation and virtual lens characteristics of the virtual camera respectively matching the location, the orientation and the lens characteristics of the projector;
align surfaces of the virtual model with the cloud of points to extract a respective location and respective orientation of the physical object relative to a physical origin of the three-dimensional environment, wherein the surfaces of the physical object are defined within the virtual model, and an actual location and orientation of the physical object in the three-dimensional environment is defined by the cloud of points, wherein aligning the surfaces of the virtual model with the cloud of points comprises comparing the surfaces of the virtual model with the cloud of points using an iterative closest point scheme;
set a virtual location and virtual orientation of the virtual model in the virtual three-dimensional environment with respect to a virtual origin, related to the physical origin, to match the respective location and the respective orientation of the physical object;
render images for projection onto the physical object based on the virtual location, the virtual orientation and the virtual lens characteristics of the virtual camera, and the respective location, the images adjusted for the virtual location and the virtual orientation of virtual model; and,
control the projector to project based on the virtual location, the virtual orientation and the virtual lens characteristics of the virtual camera and, control the projector to illuminate the physical object with the images adjusted for the virtual location and the virtual orientation of the virtual model.

2. The system of claim 1 , wherein the physical object of the three-dimensional environment comprises one or more of: a surface; a curved surface; and an irregular surface.

3. The system of claim 1 , wherein the one or more structured light patterns comprises a sequence of structured light patterns configured to uniquely sequentially illuminate different portions of the three-dimensional environment and the physical object.

4. The system of claim 1 , wherein the one or more structured light patterns comprise a sequence of images of horizontal bars and vertical bars, the bars either halving or doubling in width within the sequence and in each direction.

5. The system of claim 1 , wherein the one or more structured light patterns comprises one or more of: a sequence of images of bars; black and white images; colored images; and images of sinusoids.

6. The system of claim 1 , wherein the computing device is further configured to determine the given positions of the at least two cameras relative to the three-dimensional environment, and set a physical origin of the three-dimensional environment by:
acquiring at least one image of a known target pattern positioned in the three-dimensional environment from each of the at least two cameras; and, comparing the at least one image of the known target pattern with a representation of the known target pattern.

7. The system of claim 1 , wherein the computing device is further configured to generate the cloud of points representing the three-dimensional environment using the two-dimensional mapping and the given positions of the at least two cameras relative to the three-dimensional environment using triangulation.

8. The system of claim 1 , wherein the computing device is further configured to determine the location, the orientation and the lens characteristics of the projector relative to the three-dimensional environment, from the cloud of points by matching a plurality of points from the cloud of points to respective areas of one or more of the structured light patterns.

9. The system of claim 1 , wherein the virtual location, the virtual orientation and the virtual lens characteristics comprises pose data comprising: a three-dimensional position of the virtual camera in the virtual three-dimensional environment with respect to a virtual origin, related to the physical origin; a pitch value; a yaw value; a roll value; and an offset value.

10. A method comprising:
controlling a projector, using a computing device, to sequentially project one or more structured light patterns configured to uniquely illuminate different portions of a three-dimensional environment and a physical object, the three-dimensional environment comprising the physical object;
acquiring one or more respective images from each of at least two cameras, using the computing device, while the projector is projecting the one or more structured light patterns, each of the one or more respective images correlated with a given respective structured light pattern, each of the projector and the at least two cameras mounted relative to the three-dimensional environment with respective fields of view at least partially overlapping a projection area of the projector on the three-dimensional environment;
generating, using the computing device, a two-dimensional mapping of the different portions of the three-dimensional environment between a projector space and a camera space by processing the respective images and correlated given respective structured light patterns;
generating, using the computing device, a cloud of points representing the three-dimensional environment using the two-dimensional mapping and given positions of the at least two cameras relative to the three-dimensional environment;
determining, using the computing device, a location, an orientation and lens characteristics of the projector relative to the three-dimensional environment, from the cloud of points;
positioning, using the computing device, a virtual camera relative to a virtual three-dimensional environment, corresponding to the three-dimensional environment, a virtual location, a virtual orientation and virtual lens characteristics of the virtual camera respectively matching the location, the orientation and the lens characteristics of the projector;
aligning surfaces of a virtual model, stored in a memory, with the clou

Assignees

• Christie Digital Systems Usa

Inventors

• Moule Kevin
• Scott Derek
• Wasilka Nick
• Fraser Ian James
• Robinson James

Classifications

• G06T7/521 Primary

  from laser ranging, e.g. using interferometry; from the projection of structured light · CPC title

• H04N13/0203

  Electricity · mapped topic

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• H04N5/74

  Projection arrangements for image reproduction, e.g. using eidophor · CPC title

• H04N13/204

  using stereoscopic image cameras (stereoscopic photography G03B35/00) · CPC title