Image processing device, image processing method, and storage medium

Patent metadata

Field	Value
Publication number	US-10269092-B2
Application number	US-201715753741-A
Country	US
Kind code	B2
Filing date	Jun 13, 2017
Priority date	Jun 17, 2016
Publication date	Apr 23, 2019
Grant date	Apr 23, 2019

Abstract

An image processing device 10 includes: a fisheye image acquisition unit 11 that acquires a fisheye image; a viewpoint compensation vector acquisition unit 12 that acquires, based on a roll angle around an optical axis and a pitch angle of the optical axis of an imaging apparatus, a viewpoint compensation vector for converting the fisheye image into an image that is obtainable by taking an image of the object from a direction parallel to a ground plane of the object; and an image generation unit 13 that sets a plurality of viewpoints for a converted fisheye image that is the fisheye image converted using the viewpoint compensation vector, the plurality of viewpoints being parallel to the ground plane, performs a distortion correction by perspective projection approximation based on a coordinate system axes of which include a line of sight from a viewpoint for each of the plurality of viewpoints, and generates a single normalized image using image elements of vertical direction, the image elements each extracted from corrected fisheye images.

First claim

The invention claimed is:

1. An image processing device comprising:
a memory that stores a set of instructions; and
at least one processor configured to execute the set of instructions to:
acquire a fisheye image in which an object is;
acquire, based on a roll angle around an optical axis and a pitch angle of the optical axis of an imaging apparatus, a viewpoint compensation vector for converting the fisheye image into an image that is obtainable by taking an image of the object from a direction parallel to a ground plane of the object, the roll angle and the pitch angle being based on a plane parallel to the ground plane of the object; and
set a plurality of viewpoints for a converted fisheye image along a horizontal direction of the converted fisheye image, the plurality of viewpoints being parallel to the ground plane, the converted fisheye image being the fisheye image converted using the viewpoint compensation vector,
perform a distortion correction by perspective projection approximation on the converted fisheye image based on a coordinate system axes of which include a line of sight from a viewpoint for each of the plurality of viewpoints that are set, and
generate a single normalized image using image elements of vertical direction, the image elements each extracted from corrected fisheye images.

2. The image processing device according to claim 1 , wherein
the at least one processor is further configured to
set the viewpoint based on an all pixel number in a horizontal direction of the normalized image to be generated,
perform, for each of the plurality of viewpoints, cutting out from the corrected fisheye image in a vertical direction after performing the distortion correction, extract slice images to each of which a line of sight from a viewpoint enters,
arrange the extracted slice images in predetermined order in a horizontal direction, and generate the single normalized image.

3. The image processing device according to claim 1 , wherein
the at least one processor is further configured to
determine, in the distortion correction by perspective projection approximation for each of the plurality of viewpoints, a distance of a projection plane to an origin in the coordinate system based on a size of the normalized image to be generated, a range of a viewing angle in a horizontal direction of the normalized image, and an aspect ratio of the normalized image.

4. The image processing device according to claim 1 , wherein
the at least one processor is further configured to
acquire, as the viewpoint compensation vector, a table describing an association between coordinates on an image which is obtainable by taking an image of the object from a direction parallel to the ground plane and coordinates on the fisheye image.

5. The image processing device according to claim 1 , wherein
the at least one processor is further configured to
acquire the roll angle around the optical axis and the pitch angle of the optical axis of the imaging apparatus from an angle sensor attached to the imaging apparatus, and generate the viewpoint compensation vector based on the roll angle and the pitch angle which are acquired.

6. An image processing method comprising:
acquiring a fisheye image in which an object is;
acquiring, based on a roll angle around an optical axis and a pitch angle of the optical axis of an imaging apparatus, a viewpoint compensation vector for converting the fisheye image into an image that is obtainable by taking an image of the object from a direction parallel to a ground plane of the object, the roll angle and the pitch angle being based on a plane parallel to the ground plane of the object; and
setting a plurality of viewpoints for a converted fisheye image along a horizontal direction of the converted fisheye image, the plurality of viewpoints being parallel to the ground plane, the converted fisheye image being the fisheye image converted using the viewpoint compensation vector,
performing a distortion correction by perspective projection approximation on the converted fisheye image based on a coordinate system axes of which include a line of sight from a viewpoint for each of the plurality of viewpoints that are set, and
performing image generation of generating a single normalized image using image elements of a vertical direction, the image elements each extracted from corrected fisheye images.

7. The image processing method according to claim 6 , wherein
the image generation includes
setting the viewpoint based on an all pixel number in a horizontal direction of the normalized image to be generated,
performing, for each of the plurality of viewpoints, cutting out from the corrected fisheye image in a vertical direction after performing the distortion correction, extracting slice images to each of which a line of sight from a viewpoint enters,
arranging the extracted slice images in predetermined order in a horizontal direction, and generating a single normalized image.

8. The image processing method according to claim 6 , wherein
the image generation includes
determining, in the distortion correction by perspective projection approximation for each of the plurality of viewpoints, a distance of a projection plane to an origin in the coordinate system is determined based on a size of the normalized image to be generated, a range of a viewing angle in a horizontal direction of the normalized image, and an aspect ratio of the normalized image.

9. The image processing method according to claim 6 , wherein
the acquiring the viewpoint compensation vector includes acquiring, as the viewpoint compensation vector, a table describing an association between coordinates on an image that is obtainable by taking an image of the object from a direction parallel to the ground plane and coordinates on the fisheye image is acquired.

10. The image processing method according to claim 6 , further comprising
acquiring the roll angle around the optical axis and the pitch angle of the optical axis of the imaging apparatus from an angle sensor attached to the imaging apparatus, and calculating the viewpoint compensation vector based on the roll angle and the pitch angle which are acquired.

11. A non-transitory computer readable storage medium storing a program that causes a computer to execute:
fisheye image acquisition processing of acquiring a fisheye image in which an object is;
viewpoint compensation vector acquisition processing of acquiring, based on a roll angle around an optical axis and a pitch angle of the optical axis of an imaging apparatus, a viewpoint compensation vector for converting the fisheye image into an image that is obtainable by taking an image of the object from a direction parallel to a ground plane of the object, the roll angle and the pitch angle being based on a plane parallel to the ground plane of the object; and
image generation processing of setting a plurality of viewpoints for a converted fisheye image along a horizontal direction of the converted fisheye, the plurality of viewpoints being parallel to the ground plane, the converted fisheye image being the fisheye image converted image using the viewpoint compensation vector,
performing a distortion correction by perspective projection approximation to the converted fisheye image based on a coordinate system axes of which include a line of sight from a viewpoint for each of the plurality of viewpoints that are set, and
generating a single normalized image using image elements of a vertical direction, the image elements each extracted from corrected fisheye images.

12. The storage medium according to claim 11 , wherein
the image generation processing
se

Assignees

• Nec Corp

Inventors

• Yoshimi Kosuke

Classifications

• H04N23/60

  Control of cameras or camera modules · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/81

  for suppressing or minimising disturbance in the image signal generation · CPC title

• H04N25/61

  the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4" · CPC title

• G02B27/0037

  with diffracting elements (G02B27/0056 takes precedence; holographic optical elements G02B5/32; zone systems G02B5/1876) · CPC title