Method and apparatus for generating panoramic image with stitching process

Patent metadata

Field	Value
Publication number	US-2018227484-A1
Application number	US-201715427504-A
Country	US
Kind code	A1
Filing date	Feb 8, 2017
Priority date	Feb 8, 2017
Publication date	Aug 9, 2018
Grant date	—

Abstract

A vertex processing device applied in an image processing system having an image capture module is disclosed. The image capture module generates camera images. The vertex processing device comprises a coefficient interpolation unit and a coordinate modifying unit. The coefficient interpolation unit generates an interpolated warping coefficient for each camera image with respect to each vertex from a vertex list based on n number of warping coefficients and its original texture coordinates in each camera image. The coordinate modifying unit calculates modified texture coordinates in each camera image for each vertex according to the interpolated warping coefficient and its original texture coordinates in each camera image. The vertex list comprises vertices with data structures that define vertex mapping between the camera images and a panoramic image. The n number of warping coefficients correspond to n number of overlap regions in the panoramic image.

First claim

What is claimed is:

1 . A vertex processing device applied in an image processing system having an image capture module that generates a plurality of camera images, comprising:
a coefficient interpolation unit for generating an interpolated warping coefficient for each camera image with respect to each vertex from a vertex list based on n number of warping coefficients and its original texture coordinates in each camera image; and a coordinate modifying unit coupled to coefficient interpolation unit for calculating modified texture coordinates in each camera image for each vertex according to the interpolated warping coefficient and its original texture coordinates in each camera image; wherein the vertex list comprises a plurality of vertices with their data structures that define a vertex mapping between the camera images and a panoramic image; and wherein the n number of warping coefficients respectively denote warping degrees for n number of overlap regions in the panoramic image and n>=1.

2 . The device according to claim 1 , wherein if n=1, the interpolated warping coefficient is equal to the single warping coefficient.

3 . The device according to claim 1 , wherein if n>=2, the coefficient interpolation unit generates the interpolated warping coefficient for each camera image in relation to a target vertex by performing interpolation operations over warping coefficients of a plurality of adjacent overlap regions with respect to the original texture coordinates of the target vertex in each camera image.

4 . The device according to claim 3 , wherein when the number of adjacent overlap regions is equal to two, the coefficient interpolation unit calculates the interpolated warping coefficient for a specified camera image according to the following equation:
C′=C 2*θ1/(θ1+θ2)+ C 1*θ2/(θ1+θ2); and
wherein C′ denotes the interpolated warping coefficient, C 1 denotes a first warping coefficient of a first overlap region from the two adjacent overlap regions, C 2 denotes a second warping coefficient of a second overlap region from the two adjacent overlap regions, θ 1 denotes a first angle between a first vector from the center of the specified camera image to the location of the first overlap region and a second vector from the center of the specified camera image to the target vertex, and θ 2 denotes a second angle between the second vector and a third vector from the center of the specified camera image to the location of the second overlap region.

5 . The device according to claim 1 , wherein the coordinate modifying unit calculates the modified texture coordinates of a target vertex for a specified camera image according to the following equation:
U 1′=( U 1− U center )* C′+U center ;V 1′=( V 1 −V center )* C′+V center .;
wherein (U 1 ′, V 1 ′) denotes the modified texture coordinates of the target vertex, (U 1 , V 1 ) denotes the original texture coordinates of the target vertex, C′ denotes the interpolated warping coefficient, and (U center , V center ) denotes texture coordinates of the center of the specified camera image.

6 . The device according to claim 5 , wherein after the specified camera image is mapped to the panoramic image, an expansion image effect is created if C′<1, and a shrink image effect is created wherein if C′>1.

7 . The device according to claim 1 , wherein the vertex list is one of an original vertex list and an original surface vertex list from a calibration device, and wherein the original vertex list is divided into a plurality of original surface vertex lists.

8 . An image processing system for receiving a first vertex list from a calibration device and generating a panoramic image, comprising:
an image capture module for generating a plurality of camera images; a vertex processing device for modifying all the texture coordinates for all vertices from the first vertex list based on n number of warping coefficients to generate a second vertex list; an image processing apparatus for selectively forming the panoramic image according to the camera images and the second vertex list, and for measuring n number of region errors for n number of overlap regions in the panoramic image; and an optimizing unit for generating the n number of warping coefficients according to the n number of region errors; wherein the vertex processing device comprises: a coefficient interpolation unit for generating an interpolated warping coefficient for each camera image with respect to each vertex from the first vertex list based on the n number of warping coefficients and its original texture coordinates of the vertex in each camera image; and a coordinate modifying unit coupled to coefficient interpolation unit for calculating modified texture coordinates in each camera image for each vertex according to the interpolated warping coefficient and its original texture coordinates in each camera image; wherein each of the first vertex list and the second vertex list comprises a plurality of vertices with their data structures that define a vertex mapping between the camera images and the panoramic image; and wherein the n number of warping coefficients respectively denote warping degrees for n number of overlap regions in the panoramic image and n>=1.

9 . The system according to claim 8 , wherein if n=1, the interpolated warping coefficient is equal to the single warping coefficient.

10 . The system according to claim 8 , wherein if n>=2, the coefficient interpolation unit generates the interpolated warping coefficient for each camera image in relation to a target vertex by performing interpolation operations over warping coefficients of a plurality of adjacent overlap regions with respect to the original texture coordinates of the target vertex in each camera image.

11 . The system according to claim 10 , wherein when the number of adjacent overlap regions is equal to two, the coefficient interpolation unit calculates the interpolated warping coefficient for a specified camera image according to the following equation:
C′=C 2*θ1/(θ1+θ2)+ C 1*θ2/(θ1+θ2); and
wherein C′ denotes the interpolated warping coefficient, C 1 denotes a first warping coefficient of a first overlap region from the two adjacent overlap regions, C 2 denotes a second warping coefficient of a second overlap region from the two adjacent overlap regions, θ 1 denotes a first angle between a first vector from the center of the specified camera image to the location of the first overlap region and a second vector from the center of the specified camera image to the target vertex, and θ 2 denotes a second angle between the second vector and a third vector from the center of the specified camera image to the location of the second overlap region.

12 . The system according to claim 8 , wherein the coordinate modifying unit calculates the modified texture coordinates of a target vertex for a specified camera image according to the following equation:
U 1′=( U 1− U center )* C′+U center ;V 1′=( V 1− V center )* C′+V center .;
wherein (U 1 ′, V 1 ′) denotes the modified texture coordinates of the target vertex, (U 1 , V 1 ) denotes the original texture coordinates of the target vertex, C′ denotes the interpolated warping coefficient, and (U center , V center ) denotes texture coordinates of the center of the specified camera image.

13 . The system according to claim 8 , wherein the optimizing unit further uses an optimization algorithm to modify the n number of warping coefficients so as to minimize the n number of region errors.

14 . The system according to claim 8 , wherein the image

Assignees

• Aspeed Technology Inc

Inventors

• HUNG Pei-Hen
• Lu Chung-Yen
• Yeh Kuo-Wei
• Chen Jing-Chuan

Classifications

• H04N23/698 Primary

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• G06T3/4007 Primary

  based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title

• G06T2200/32

  involving image mosaicing · CPC title

• H04N5/23238 Primary

  Electricity · mapped topic