Image processing apparatus and image processing method to increase encoding efficiency of two-dimensional image

Patent metadata

Field	Value
Publication number	US-11317114-B2
Application number	US-201816979660-A
Country	US
Kind code	B2
Filing date	Dec 11, 2018
Priority date	Mar 19, 2018
Publication date	Apr 26, 2022
Grant date	Apr 26, 2022

Abstract

A padding processing section replaces, by using an input image which is a two-dimensional image in which plural fisheye images configuring a predetermined azimuth range are provided side by side, an image to be used in a prediction process outside a valid region of a fisheye image with an image at a corresponding position in a valid region of another fisheye image based on continuity of the plural fisheye images to reduce prediction errors. The padding processing section calculates the position in the valid region of the other fisheye image corresponding to the position outside the valid region based on fisheye image information indicative, for each fisheye image, of a center position, a fisheye radius, or a state of reversal or rotation of the fisheye image in the two-dimensional image. A reversible encoding section includes the fisheye image information into an encoded stream of the two-dimensional image.

First claim

The invention claimed is:

1. An image processing apparatus, comprising:
a padding processing section configured to perform, by using a two-dimensional image in which plural fisheye images configuring a determined azimuth range are provided side by side, a padding process to replace an image at a position outside a valid region of a first fisheye image with an image at a corresponding position in a valid region of a second fisheye image based on continuity of the plural fisheye images, wherein
based on the position outside the valid region of the first fisheye image that corresponds to positions included in each of the plural fisheye images, the padding processing section is further configured to perform replacement using a fisheye image of the plural fisheye images in which a distance from a fisheye center of the fisheye image that includes the corresponding position in the valid region of the plural fisheye images to the position outside the valid region of the first fisheye image for which the padding process is to be performed is shortest.

2. The image processing apparatus according to claim 1 , wherein
the padding processing section is further configured to replace an image at a position to be used in a prediction process outside the valid region of the first fisheye image with an image at the corresponding position in the valid region of the second fisheye image.

3. The image processing apparatus according to claim 1 , wherein
the valid region of the first fisheye image and the valid region of the second fisheye image is a region within a fisheye radius.

4. The image processing apparatus according to claim 3 , wherein
the padding processing section is further configured to calculate the position in the valid region of the second fisheye image corresponding to the position outside the valid region based on the fisheye radius and an arrangement state of the plural fisheye images.

5. The image processing apparatus according to claim 3 , wherein
the padding processing section is further configured to calculate the position in the valid region of the second fisheye image corresponding to the position outside the valid region based on reversal or rotation for each fisheye image of the plural fisheye images.

6. The image processing apparatus according to claim 1 , further comprising:
an encoding section configured to perform a prediction encoding process of the two-dimensional image to generate an encoded stream, wherein
the encoding section is further configured to perform the prediction encoding process based on an image for which the padding process has been performed.

7. The image processing apparatus according to claim 6 , wherein
the encoding section is further configured to perform a quantization process and a prediction process based on the two-dimensional image for which the padding process has been performed.

8. The image processing apparatus according to claim 7 , wherein
the encoding section is further configured to perform the padding process for an image that is an adjacent block of an intra prediction process outside the valid region of the first fisheye image or an image that is a search range for a motion vector in an inter prediction process.

9. The image processing apparatus according to claim 6 , wherein
the encoding section is further configured to perform a prediction process using the two-dimensional image before the padding process is performed, and
the padding processing section is further configured to perform the padding process based on an image of an adjacent block to a block for which an intra prediction process is to be performed or an image of a search range for a motion vector in a block for which an inter prediction process is to be performed is outside the valid region.

10. The image processing apparatus according to claim 6 , wherein
the encoding section includes fisheye image information related to the plural fisheye images provided in the two-dimensional image into the encoded stream.

11. The image processing apparatus according to claim 10 , wherein
the fisheye image information includes information indicative, for each of the plural fisheye images, of a center position of each of the plural fisheye images in the two-dimensional image, a fisheye radius, and a state of reversal or rotation for each fisheye image of the plural fisheye images.

12. The image processing apparatus according to claim 1 , further comprising:
a decoding section configured to:
decode an encoded stream generated based on a prediction encoding process of the two-dimensional image; and
replace, where an image to be used in a prediction process is at a position outside the valid region of the first fisheye image, the image to be used in the prediction process with a decoded image at a corresponding position in the valid region of the second fisheye image.

13. The image processing apparatus according to claim 12 , wherein
the image to be used in the prediction process includes an image of an adjacent block to a block for which an intra prediction process is to be performed or an image of a reference block based on motion vector information of a block for which motion compensation is to be performed and reference frame information.

14. The image processing apparatus according to claim 12 , wherein
the encoded stream includes fisheye image information relating to the plural fisheye images provided in the two-dimensional image, and
the decoding section is further configured to calculate the corresponding position in the valid region of the second fisheye image based on the fisheye image information obtained based on the decoded encoded stream.

15. The image processing apparatus according to claim 12 , wherein
the decoding section is further configured to replace an image outside the valid region of the plural fisheye images in the two-dimensional image after decode with an invalid image.

16. An image processing method, comprising:
replacing in a padding processing section, by using a two-dimensional image in which plural fisheye images configuring a determined azimuth range are provided side by side, an image at a position outside a valid region of a first fisheye image with an image at a corresponding position in a valid region of a second fisheye image based on continuity of the plural fisheye images, wherein
based on the position outside the valid region of the first fisheye image corresponding to positions included in each of the plural fisheye images, the replacing in the padding processing section includes replacement using a fisheye image of the plural fisheye images in which a distance from a fisheye center of the fisheye image that includes the corresponding position in the valid region of the plural fisheye images to the position outside the valid region of the first fisheye image is shortest.

Assignees

• Sony Corp

Inventors

• Kim Jongdae

Classifications

• H04N19/55

  Motion estimation with spatial constraints, e.g. at image or region borders · CPC title

• H04N19/563 Primary

  Motion estimation with padding, i.e. with filling of non-object values in an arbitrarily shaped picture block or region for estimation purposes · CPC title

• H04N19/85

  using pre-processing or post-processing specially adapted for video compression · CPC title

• H04N19/176

  the region being a block, e.g. a macroblock · CPC title

• H04N19/52

  by predictive encoding · CPC title