Image processing device and image processing method

Patent metadata

Field	Value
Publication number	US-2016364853-A1
Application number	US-201615138644-A
Country	US
Kind code	A1
Filing date	Apr 26, 2016
Priority date	Jun 10, 2015
Publication date	Dec 15, 2016
Grant date	—

Abstract

A device includes a memory configured to store map point information including a first map point group indicating a three-dimensional position of an object measured by a distance sensor and a second map point group indicating a three-dimensional position of the object determined based on captured images and a processor configured to compare a first projection point group of the first map point group with a feature point group extracted from a new captured image, create pairs of a feature point in the feature point group and a first projection point that is similar to the feature point, when the number of the pairs is less than or equal to a threshold, compare a second projection point group of the second map point group with the feature point group, create other pairs of the feature point and a second projection point, and estimate a position and attitude of a camera.

First claim

What is claimed is:

1 . An image processing device comprising:
a memory configured to store map point information including a first map point group indicating a three-dimensional position of an object measured by a distance sensor and a second map point group indicating a three-dimensional position of the object determined based on a plurality of captured images; and a processor coupled to the memory and configured to:
acquire a new captured image including the object,
compare a first projection point group with a feature point group extracted from the new captured image, the first projection point group being obtained by projecting the first map point group on the new captured image,
create pairs of a feature point in the feature point group and a first projection point that is similar to the feature point,
when the number of the pairs is less than or equal to a threshold, compare a second projection point group with the feature point group, the second projection point group being obtained by projecting the second map point group on the new captured image,
create other pairs of the feature point and a second projection point that is similar to the feature point, and
estimate a position and attitude of a camera that has captured the new captured image based on the pairs, the other pairs, and the map point information.

2 . The image processing device according to claim 1 , wherein the map point information is generated by prioritizing information measured by the distance sensor to information determined based on the plurality of captured images.

3 . The image processing device according to claim 1 , wherein the processor is configured to create the other pairs by prioritizing second map points closer to the camera in the second map point group when the number of the pairs is less than or equal to the threshold.

4 . The image processing device according to claim 1 , wherein for an area where distribution of the first projection point group is less, the processor is configured to select the second map point group projected on the area so that the first projection point group and the second projection point group are distributed in a wide range on the new captured image.

5 . The image processing device according to claim 1 , wherein object information is displayed on the new captured image based on the result.

6 . The image processing device according to claim 5 , wherein the object information is an AR content.

7 . An image processing method executed by a computer, the image processing method comprising:
storing, into a memory, map point information including a first map point group indicating a three-dimensional position of an object measured by a distance sensor and a second map point group indicating a three-dimensional position of the object determined based on a plurality of captured images; acquiring a new captured image including the object; comparing a first projection point group with a feature point group extracted from the new captured image, the first projection point group being obtained by projecting the first map point group on the new captured image; creating pairs of a feature point in the feature point group and a first projection point that is similar to the feature point; when the number of the pairs is less than or equal to a threshold, comparing a second projection point group with the feature point group, the second projection point group being obtained by projecting the second map point group on the new captured image; creating other pairs of the feature point and a second projection point that is similar to the feature point; and estimating a position and attitude of a camera that has captured the new captured image based on the pairs, the other pairs, and the map point information.

8 . The image processing method according to claim 7 , wherein the map point information is generated by prioritizing information measured by the distance sensor to information determined based on the plurality of captured images.

9 . The image processing method according to claim 7 , wherein the other pairs is created by prioritizing second map points closer to the camera in the second map point group when the number of the pairs is less than or equal to the threshold.

10 . The image processing method according to claim 7 , wherein for an area where distribution of the first projection point group is less, the second map point group projected on the area is selected so that the first projection point group and the second projection point group are distributed in a wide range on the new captured image.

11 . The image processing method according to claim 7 , wherein object information is displayed on the new captured image based on the result.

12 . The image processing method according to claim 11 , wherein the object information is an AR content.

13 . A non-transitory computer-readable storage medium storing an image processing program for causing a computer to execute a process, the process comprising:
storing, into a memory, map point information including a first map point group indicating a three-dimensional position of an object measured by a distance sensor and a second map point group indicating a three-dimensional position of the object determined based on a plurality of captured images; acquiring a new captured image including the object; comparing a first projection point group with a feature point group extracted from the new captured image, the first projection point group being obtained by projecting the first map point group on the new captured image; creating pairs of a feature point in the feature point group and a first projection point that is similar to the feature point; when the number of the pairs is less than or equal to a threshold, comparing a second projection point group with the feature point group, the second projection point group being obtained by projecting the second map point group on the new captured image; creating other pairs of the feature point and a second projection point that is similar to the feature point; and estimating a position and attitude of a camera that has captured the new captured image based on the pairs, the other pairs, and the map point information.

14 . The non-transitory computer-readable storage medium according to claim 13 , wherein the map point information is generated by prioritizing information measured by the distance sensor to information determined based on the plurality of captured images.

15 . The non-transitory computer-readable storage medium according to claim 13 , wherein the other pairs is created by prioritizing second map points closer to the camera in the second map point group when the number of the pairs is less than or equal to the threshold.

16 . The non-transitory computer-readable storage medium according to claim 13 , wherein for an area where distribution of the first projection point group is less, the second map point group projected on the area is selected so that the first projection point group and the second projection point group are distributed in a wide range on the new captured image.

17 . The non-transitory computer-readable storage medium according to claim 13 , wherein object information is displayed on the new captured image based on the result.

18 . The non-transitory computer-readable storage medium according to claim 17 , wherein the object information is an AR content.

Assignees

• Fujitsu Ltd

Inventors

• Yamaguchi Nobuyasu

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06K9/00671

  Physics · mapped topic

• G06T7/004 Primary

  Physics · mapped topic

• G06T7/0022

  Physics · mapped topic

• G06T2207/30244

  Camera pose · CPC title