Method of and system for projecting digital information on a real object in a real environment

Patent metadata

Field	Value
Publication number	US-11652965-B2
Application number	US-202017028360-A
Country	US
Kind code	B2
Filing date	Sep 22, 2020
Priority date	Dec 28, 2012
Publication date	May 16, 2023
Grant date	May 16, 2023

Abstract

A method of projecting digital information on a real object in a real environment includes the steps of projecting digital information on a real object or part of a real object with a visible light projector, capturing at least one image of the real object with the projected digital information using a camera, providing a depth sensor registered with the camera, the depth sensor capturing depth data of the real object or part of the real object, and calculating a spatial transformation between the visible light projector and the real object based on the at least one image and the depth data. The invention is also concerned with a corresponding system.

First claim

The invention claimed is:

1. A method of projecting digital information on a real object in a real environment, comprising:
capturing, by a visible light camera, an image of a real object onto which digital information is projected by a projector;
capturing, by a depth sensor, depth data for the real object at one or more locations on the real object at which the digital information is projected, wherein the depth sensor and the camera are interrelated in a device;
calculating a spatial transformation between the projector and the real object based on the digital information in the image and the depth data corresponding to the projected digital information; and
determining calibration data of the projector based on the calculated spatial transformation.

2. The method of claim 1 , wherein the digital information comprises a visual pattern having a known geometry.

3. The method of claim 2 , further comprising:
estimating intrinsic parameters of the projector based on first coordinates of the visual pattern in a projector coordinate system and second coordinates of the visual pattern in a camera coordinate system.

4. The method of claim 3 , further comprising:
estimating a spatial transformation between the visible light camera and the projector based on the estimated intrinsic parameters,
wherein the spatial transformation between the projector and the real object is further calculated based on the estimated spatial transformation between the visible light camera and the projector.

5. The method of claim 4 , further comprising:
estimating a transformation between the projector and the real object based on the spatial transformation between the projector and the real object and the estimated spatial transformation between the visible light camera and the projector.

6. The method of claim 1 , wherein the device is a head mounted device.

7. The method of claim 1 , wherein the projector and the visible light camera are not physically coupled.

8. A non-transitory computer readable medium comprising computer readable code projecting digital information on a real object in a real environment, the computer readable code executable by one or more processors to:
capture, by a visible light camera, an image of a real object onto which digital information is projected by a projector;
capture, by a depth sensor, depth data for the real object at one or more locations on the real object at which the digital information is projected, wherein the depth sensor and the camera are interrelated in a device;
calculate a spatial transformation between the projector and the real object based on the digital information in the image and the depth data corresponding to the projected digital information; and
determine calibration data of the projector based on the calculated spatial transformation.

9. The non-transitory computer readable medium of claim 8 , wherein the digital information comprises a visual pattern having a known geometry.

10. The non-transitory computer readable medium of claim 9 , further comprising computer readable code to:
estimate intrinsic parameters of the projector based on first coordinates of the visual pattern in a projector coordinate system and second coordinates of the visual pattern in a camera coordinate system.

11. The non-transitory computer readable medium of claim 10 , further comprising computer readable code to:
estimate a spatial transformation between the visible light camera and the projector based on the estimated intrinsic parameters,
wherein the spatial transformation between the projector and the real object is further calculated based on the estimated spatial transformation between the visible light camera and the projector.

12. The non-transitory computer readable medium of claim 11 , further comprising computer readable code to:
estimate a transformation between the projector and the real object based on the spatial transformation between the projector and the real object and the estimated spatial transformation between the visible light camera and the projector.

13. The non-transitory computer readable medium of claim 8 , wherein the device is a head mounted device.

14. The non-transitory computer readable medium of claim 8 , wherein the projector and the visible light camera are not physically coupled.

15. A system for projecting digital information on a real object in a real environment, comprising:
one or more processors; and
one or more computer readable media comprising computer readable code executable by one or more processors to:
capture, by a visible light camera, an image of a real object onto which digital information is projected by a projector;
capture, by a depth sensor, depth data for the real object at one or more locations on the real object at which the digital information is projected, wherein the depth sensor and the camera are interrelated in a device;
calculate a spatial transformation between the projector and the real object based on the digital information in the image and the depth data corresponding to the projected digital information; and
determine calibration data of the projector based on the calculated spatial transformation.

16. The system of claim 15 , wherein the digital information comprises a visual pattern having a known geometry.

17. The system of claim 16 , further comprising computer readable code to:
estimate intrinsic parameters of the projector based on first coordinates of the visual pattern in a projector coordinate system and second coordinates of the visual pattern in a camera coordinate system.

18. The system of claim 17 , further comprising computer readable code to:
estimate a spatial transformation between the visible light camera and the projector based on the estimated intrinsic parameters,
wherein the spatial transformation between the projector and the real object is further calculated based on the estimated spatial transformation between the visible light camera and the projector.

19. The system of claim 15 , wherein the visible light camera and the depth sensor are comprised in a head mounted device.

20. The system of claim 15 , wherein the projector and the visible light camera are not physically coupled.

Assignees

• Apple Inc

Inventors

• Meier Peter
• Ben Himane Mohamed Selim
• Kurz Daniel

Classifications

• G01B11/25

  by projecting a pattern, e.g. {one or more lines,} moiré fringes on the object (G01B11/255 takes precedence {; image analysis for depth or shape recovery G06T7/50}) · CPC title

• H04N13/257

  Colour aspects · CPC title

• H04N9/3191

  Testing thereof (testing of displays in general G09G3/006) · CPC title

• H04N9/3185 Primary

  Geometric adjustment, e.g. keystone or convergence (optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14) · CPC title

• H04N13/128

  Adjusting depth or disparity · CPC title