Image processing device and projection system

Patent metadata

Field	Value
Publication number	US-10929706-B2
Application number	US-201916292552-A
Country	US
Kind code	B2
Filing date	Mar 5, 2019
Priority date	Mar 19, 2018
Publication date	Feb 23, 2021
Grant date	Feb 23, 2021

Abstract

An image processing device extracts, for a target area that has sites having different flexibilities and where markers corresponding to the respective flexibilities are put on the sites, marking positions where the markers are put according to each type of the markers from each of a first image to be projected and being an image of the target area captured in advance and a second image being an image of the target area captured before projection of the first image; executes, based on the extracted marking positions according to each type of the markers, image processing to align the marking positions of the markers contained in the first image with the marking positions of the markers contained in the second image by a processing method corresponding to the types of the markers; and generates, based on the processed first image, a projection image to be projected onto the target area.

First claim

What is claimed is:

1. An image processing device comprising:
a marking information extractor to, for a target area including a plurality of sites having different flexibilities, a plurality of markers of types corresponding to the respective flexibilities being put on the plurality of sites, extract marking positions, the plurality of markers being put in the marking positions according to each type of the plurality of markers, from each of a first image, that is to be projected and that is an image of the target area captured in advance, and a second image that is an image of the target area captured before projection of the first image;
an image processor to, based on the marking positions extracted according to each of the types of the plurality of markers, execute image processing to align the marking positions of the markers contained in the first image with the marking positions of the markers contained in the second image by a processing method corresponding to the types of the plurality of markers; and
a projection image generator to, based on the first image after the image processing, generate a projection image to be projected onto the target area,
wherein the image processor includes:
an alignment processor to, based on the plurality of marking positions of first type markers put on a non-varying site with relatively low flexibility, perform by linear transformation, the image processing to align the plurality of marking positions of the first type markers contained in the first image with the plurality of marking positions of the first type markers contained in the second image; and
a deformation processor to, based on the plurality of marking positions of second type markers put on a varying site with relatively higher flexibility than flexibility of the non-varying site, perform by non-linear transformation, the image processing to align the plurality of marking positions of the second type markers contained in the first image and the plurality of marking positions of the second type markers contained in the second image.

2. The image processing device of claim 1 , further comprising a shape acquisition unit to acquire a shape of a surface of the target area from the second image,
wherein
the alignment processing unit is configured to perform the liner transformation on a whole of the first image such that the plurality of marking positions of the second type markers contained in the first image are marking positions on the shape of the surface corresponding to the plurality of marking positions of the first type markers extracted from the second image, and
the deformation process unit is configured to perform the non-liner transformation on part of the first mage after the liner-transformation such that the plurality of marking positions of the second type markers contained in the first image after the liner transformation are marking positions on the shape of the surface corresponding to the plurality of marking positions of the second type markers extracted from the second image.

3. The image processing device of claim 1 , wherein
the alignment processing unit is configured to execute alignment of the first type markers covering three-dimensional rotation.

4. The image processing device of claim 1 , wherein
the alignment processing unit is configured to make a notification of an error of transformation of the first type markers, after alignment exceeds a threshold.

5. The image processing device of claim 1 , further comprising:
a display unit to display at least one of the first image and the second image; and
an operator, to receive a marking operation of marking the image displayed on the display unit with the markers whose marker type, is specified,
wherein the marking information extractor is configured to extract the marking positions of the plurality of markers put by the marking operation according to each of the types of the plurality of markers.

6. A projection system comprising:
an input unit to, for a target area including a plurality of sites having different flexibilities, a plurality of markers of types corresponding to the respective flexibilities being put on the plurality of sites, input a first image that is to be projected and that is an image of the target area;
an image capturing unit to capture an image of the target area to generate a second image;
a marking information extractor to extract marking positions, the plurality of markers being put in the marking positions according to each type of the plurality of markers from each of the first image and the second image;
an image processor to, based on the marking positions extracted according to each of the types of the plurality of markers, execute image processing to align the marking positions of the plurality of markers put on the first image with the marking positions of the plurality of markers put on the second image, by a processing method corresponding to the types of the plurality of markers;
a projection image generator to, based on the first image after the image processing, generate a projection image to be projected onto the target area; and
a projection unit to project the projection image onto the target area,
wherein the image processor includes:
an alignment processor to, based on the plurality of marking positions of first type markers put on a non-varying site with relatively low flexibility, perform by linear transformation, the image processing to align the plurality of marking positions of the first type markers contained in the first image with the plurality of marking positions of the first type markers contained in the second image; and
a deformation processor to, based on the plurality of marking positions of second type markers put on a varying site with relatively higher flexibility than flexibility of the non-varying site, perform by non-linear transformation, the image processing to align the plurality of marking positions of the second type markers contained in the first image and the plurality of marking positions of the second type markers contained in the second image.

7. The projection system of claim 6 , further comprising:
a shape acquisition unit to acquire a shape of a surface of the target area from the second image, wherein
the alignment processor is configured to perform the liner transformation on a whole of the first image such that the plurality of marking positions of the second type markers contained in the first image are marking positions on the shape of the surface corresponding to the plurality of marking positions of the first type markers extracted from the second image, and
the deformation processor is configured to perform the non-liner transformation on part of the first mage after the liner-transformation such that the plurality of marking positions of the second type markers contained in the first image after the liner transformation are marking positions on the shape of the surface corresponding to the plurality of marking positions of the second type markers extracted from the second image.

8. The projection system of claim 6 , wherein
the alignment processor is configured to execute alignment of the first type markers covering three-dimensional rotation.

9. The projection system of claim 6 , wherein the alignment processor is configured to make a notification of an error of transformation of the first type markers, after alignment exceeds a threshold.

10. The projection system of claim 6 , further comprising:
a display to display at least one of the first image and the second image; and
an operator, to receive a marking operation of marking the image displayed on the display unit with the markers whose marker type, is specified,
wh

Assignees

• Nakata Otoichi
• Ricoh Co Ltd

Inventors

• Nakata Otoichi

Classifications

• G06T7/33 Primary

  using feature-based methods · CPC title

• G06T2207/30004

  Biomedical image processing · CPC title

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• G06T2207/10072

  Tomographic images · CPC title

• H04N9/3185

  Geometric adjustment, e.g. keystone or convergence (optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14) · CPC title