Portable projection mapping device and projection mapping system

Patent metadata

Field	Value
Publication number	US-11758100-B2
Application number	US-202017753074-A
Country	US
Kind code	B2
Filing date	Sep 9, 2020
Priority date	Sep 11, 2019
Publication date	Sep 12, 2023
Grant date	Sep 12, 2023

Abstract

A device may provide, to a camera and a projector of a portable projection mapping device, first instructions for calibrating the camera and the projector, and may receive, based on the first instructions, calibration parameters for the camera and the projector. The device may calculate a stereo calibration between the camera and the projector based on the calibration parameters, and may provide, to the camera, second instructions for recognizing a reference instrument associated with the portable projection mapping device. The device may receive, based on the second instructions, binocular images, and may determine additional parameters based on the binocular images. The device may determine recognition parameters for recognizing the reference instrument, based on the binocular images and the additional parameters. The device may process the recognition parameters and the stereo calibration, with an optical tracking model, to generate and provide overlay visualization data to the portable projection mapping device.

First claim

What is claimed is:

1. A method comprising:
providing, by a device and to a camera and a projector of a portable projection mapping device, first instructions for calibrating the camera and the projector;
receiving, by the device, from the camera and the projector, and based on the first instructions, calibration parameters for the camera and the projector;
calculating, by the device, a stereo calibration between the camera and the projector based on the calibration parameters for the camera and the projector;
providing, by the device and to the camera, second instructions for recognizing a reference instrument and a digitizing probe associated with the portable projection mapping device;
receiving, by the device, from the camera, and based on the second instructions, binocular images;
determining, by the device, a marker extraction, a marker matching, a three-dimensional coordinate reconstruction of markers, a model fitting, and a three-dimensional registration, based on the binocular images;
determining, by the device, recognition parameters for recognizing the reference instrument and the digitizing probe, based on the binocular images, the marker extraction, the marker matching, the three-dimensional coordinate reconstruction of markers, the model fitting, and the three-dimensional registration;
processing, by the device, the recognition parameters for recognizing the reference instrument and the stereo calibration between the camera and the projector, with an optical tracking model, to generate overlay visualization data; and
providing, by the device and to the portable projection mapping device, the overlay visualization data.

2. The method of claim 1 , wherein the overlay visualization data cause the portable projection mapping device to project a visualization on a target object.

3. The method of claim 2 , further comprising:
utilizing the recognition parameters for recognizing the digitizing probe to enable a user to interact with the visualization.

4. The method of claim 2 , wherein the visualization includes one of an image or a video.

5. The method of claim 1 , wherein the first instructions cause the camera to generate intrinsic camera calibration parameters and extrinsic camera calibration parameters and cause the projector to generate intrinsic projector calibration parameters,
wherein the calibration parameters for the camera and the projector include the intrinsic camera calibration parameters, the extrinsic camera calibration parameters, and the intrinsic projector calibration parameters.

6. The method of claim 1 , wherein calculating the stereo calibration between the camera and the projector based on the calibration parameters for the camera and the projector comprises:
processing the calibration parameters for the camera and the projector, with a computer vision model, to calculate the stereo calibration between the camera and the projector.

7. The method of claim 1 , further comprising:
processing the binocular images, with an optical tracking callback model, to track markers of the digitizing probe.

8. A device, comprising:
one or more memories; and
one or more processors, communicatively coupled to the one or more memories, configured to:
provide, to a camera and a projector of a portable projection mapping device, first instructions for calibrating the camera and the projector;
receive, from the camera and the projector and based on the first instructions, calibration parameters for the camera and the projector;
calculate a stereo calibration between the camera and the projector based on the calibration parameters for the camera and the projector;
provide, to the camera, second instructions for recognizing a reference instrument and a digitizing probe associated with the portable projection mapping device;
receive, from the camera and based on the second instructions, binocular images;
determine one or more of a marker extraction, a marker matching, a three-dimensional coordinate reconstruction of markers, a model fitting, or a three-dimensional registration, based on the binocular images;
determine recognition parameters for recognizing the reference instrument and the digitizing probe, based on the binocular images, the marker extraction, the marker matching, the three-dimensional coordinate reconstruction of markers, the model fitting, and the three-dimensional registration;
process the recognition parameters for recognizing the reference instrument and the stereo calibration between the camera and the projector, with an optical tracking model, to generate overlay visualization data;
provide, to the portable projection mapping device, the overlay visualization data,
wherein the overlay visualization data cause the portable projection mapping device to project a visualization on a target object; and
utilize the recognition parameters for recognizing the digitizing probe to enable a user to interact with the visualization.

9. The device of claim 8 , wherein the one or more processors, when determining the marker extraction, are configured to:
process the marker extraction, with a computer vision model, to optimize performance of the marker extraction for different scenarios.

10. The device of claim 8 , wherein the one or more processors, when determining the marker matching, are configured to:
sort detected markers in the binocular images according to vertical positions of the detected markers and horizontal positions of the detected markers; and
match the detected markers, to generate the marker matching, based on sorting the detected markers.

11. The device of claim 8 , wherein the one or more processors, when determining the three-dimensional coordinate reconstruction of markers, are configured to:
utilize triangulation on detected markers in the binocular images to determine the three-dimensional coordinate reconstruction of markers.

12. The device of claim 8 , wherein the one or more processors, when determining the model fitting, are configured to:
match a subset of the three-dimensional coordinate reconstruction of markers to detected markers in the binocular images to determine the model fitting.

13. The device of claim 8 , wherein the one or more processors, when determining the three-dimensional registration, are configured to:
calculate a singular-value decomposition of a derived matrix associated with the model fitting to determine the three-dimensional registration.

14. The device of claim 8 , wherein the overlay visualization data is associated with an augmented reality image or video.

15. A non-transitory computer-readable medium storing a set of instructions, the set of instructions comprising:
one or more instructions that, when executed by one or more processors of a device, cause the device to:
provide, to a camera and a projector of a portable projection mapping device, first instructions for calibrating the camera and the projector;
receive, from the camera and the projector and based on the first instructions, calibration parameters for the camera and the projector;
calculate a stereo calibration between the camera and the projector based on the calibration parameters for the camera and the projector;
provide, to the camera, second instructions for recognizing a reference instrument and a digitizing probe associated with the portable projection mapping device;
receive, from the camera and based on the second instructions, binocular images;
determine a marker extraction, a marker matching, a three-dimensional coordinate reconstruction of markers, a model fitting, and a three-dimensional registration, based on the bi

Assignees

• Univ Johns Hopkins

Inventors

• Armand Mehran
• Liu Shuya
• Huang Wei-Lun
• Shin Austin

Classifications

• H04N9/3185 Primary

  Geometric adjustment, e.g. keystone or convergence (optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14) · CPC title

• G06T7/80

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title

• G06T2207/30204

  Marker · CPC title

• H04N9/3194

  including sensor feedback · CPC title

• G03B17/54

  with projector · CPC title