System and method for fisheye image processing

Patent metadata

Field	Value
Publication number	US-11935210-B2
Application number	US-202017018627-A
Country	US
Kind code	B2
Filing date	Sep 11, 2020
Priority date	Oct 19, 2018
Publication date	Mar 19, 2024
Grant date	Mar 19, 2024

Abstract

A system and method for fisheye image processing can be configured to: receive fisheye image data from at least one fisheye lens camera associated with an autonomous vehicle, the fisheye image data representing at least one fisheye image frame; partition the fisheye image frame into a plurality of image portions representing portions of the fisheye image frame; warp each of the plurality of image portions to map an arc of a camera projected view into a line corresponding to a mapped target view, the mapped target view being generally orthogonal to a line between a camera center and a center of the arc of the camera projected view; combine the plurality of warped image portions to form a combined resulting fisheye image data set representing recovered or distortion-reduced fisheye image data corresponding to the fisheye image frame; generate auto-calibration data representing a correspondence between pixels in the at least one fisheye image frame and corresponding pixels in the combined resulting fisheye image data set; and provide the combined resulting fisheye image data set as an output for other autonomous vehicle subsystems.

First claim

What is claimed is:

1. A system comprising:
a data processor; and
a fisheye image processing system, executable by the data processor, the fisheye image processing system comprising an offline processing stage and an online processing stage, wherein the fisheye image processing system is configured to:
receive fisheye image data from at least one fisheye lens camera associated with an autonomous vehicle, the fisheye image data representing at least one fisheye image frame;
use the offline processing stage to pre-process the fisheye image data to generate one or more combined image transformations;
use the combined image transformations as an input to the online processing stage;
quickly remove distortion from each raw fisheye image frame in a fisheye image data set provided to the online processing stage using the combined image transformations; and
provide the combined resulting fisheye image data set as an output for other autonomous vehicle subsystems;
wherein the online processing stage is configured to:

partition the raw fisheye image data into a second plurality of image portions representing portions of the fisheye image frame;
warp each of the second plurality of image portions to map an arc of a camera projected view into a line corresponding to a mapped target view, the warping comprising taking auto-calibration data as an input in addition to the second plurality of image portions;
produce distortion-reduced fisheye image data; and
provide the distortion-reduced fisheye image data to an autonomous vehicle control system for further processing.

2. The system of claim 1 , wherein the offline processing stage is configured to:
partition each of the at least one fisheye image frame of the fisheye image data into a plurality of image portions representing portions of each fisheye image frame;
warp each of the plurality of image portions to map an arc of a camera projected view into a line corresponding to a mapped target view, the mapped target view being generally orthogonal to a line between a camera center and a center of the arc of the camera projected view; and
of image portions to form the combined image transformations.

3. The system of claim 1 , wherein the combined image transformations are combined into a single combine transformation, further wherein the single combined transformation is applied by the online processing stage once.

4. The system of claim 2 , wherein the combined image transformations are combined into a single combine transformation, further wherein the single combined transformation is applied by the online processing stage once.

5. The system of claim 1 wherein the at least one fisheye lens camera is from the group consisting of: a left side fisheye lens camera, a right side fisheye lens camera, and a top mounted fisheye lens camera.

6. The system of claim 1 being further configured to obtain parameters of the fisheye lens camera, the parameters including a fisheye lens radius, a lens aperture, a focal length, and a target field of view angle.

7. The system of claim 1 being further configured to map pixels in the fisheye image frame to pixels in the combined resulting fisheye image data set.

8. The system of claim 1 being further configured reduce redundant computation time for each image, so that the system reaches a real-time performance of 50 Hz per image.

9. A method comprising:
receiving fisheye image data from at least one fisheye lens camera associated with an autonomous vehicle, the fisheye image data representing at least one fisheye image frame;
partitioning each frame of the at least one fisheye image frame into a plurality of image portions representing portions of the fisheye image frame;
warping each of the plurality of image portions to map an arc of a camera projected view into a line corresponding to a mapped target view, the mapped target view being generally orthogonal to a line between a camera center and a center of the arc of the camera projected view;
processing the warped plurality of image portions to produce a first transformation;
stitching together each of the warped plurality of image portions to form a combined resulting fisheye image data set;
processing the combined resulting fisheye image data set to produce a second transformation;
generating auto-calibration data representing a combination of the first transformation and the second transformation;
providing the auto-calibration data to an online processing stage of an imaging processing module;
providing raw fisheye image data from the at least one fisheye lens camera associated with an autonomous vehicle to the online processing stage;
partitioning, by the online processing stage, the raw fisheye image data into a second plurality of image portions representing portions of the fisheye image frame;
warping, by the online processing stage, each of the second plurality of image portions to map the arc of the camera projected view into a line corresponding to a mapped target view, the warping comprising taking the auto-calibration data as an input in addition to the second plurality of image portions;
producing distortion-reduced fisheye image data; and
providing the distortion-reduced fisheye image data to an autonomous vehicle control system for further processing.

10. The method of claim 9 including performing object extraction on the combined resulting fisheye image data set to identify extracted objects in the fisheye image frame.

11. The method of claim 9 including aligning the mapped target view with a line parallel to a side of the autonomous vehicle.

12. The method of claim 9 wherein the at least one fisheye lens camera is from the group consisting of: a left side fisheye lens camera, a right side fisheye lens camera, and a top mounted fisheye lens camera.

13. The method of claim 9 including obtaining parameters of the fisheye lens camera, the parameters including a fisheye lens radius, a lens aperture, a focal length, and a target field of view angle.

14. The method of claim 9 including mapping pixels in the fisheye image frame to pixels in the combined resulting fisheye image data set.

15. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:
receive fisheye image data from at least one fisheye lens camera associated with an autonomous vehicle, the fisheye image data representing at least one fisheye image frame;
partition each frame of the at least one fisheye image frame into a plurality of image portions representing portions of the fisheye image frame;
warp each of the plurality of image portions to map an arc of a camera projected view into a line corresponding to a mapped target view, the mapped target view being generally orthogonal to a line between a camera center and a center of the arc of the camera projected view;
process the warped plurality of image portions to produce a first transformation;
stitch together each of the warped plurality of image portions to form a combined resulting fisheye image data set;
process the combined resulting fisheye image data set to produce a second transformation;
generate auto-calibration data representing a combination of the first transformation and the second transformation;
provide the auto-calibration data to an online processing stage of an imaging processing module;
provide raw fisheye image data from the at least one fisheye lens camera associated with an autonomous vehicle to the online processing stage;
partition, by the online processing stage, the raw fisheye image data into a second plurality of image po

Assignees

• Tusimple Inc

Inventors

• Yan Zhipeng
• Chen Pengfei
• Wang Panqu

Classifications

• G06T3/0018 Primary

  Physics · mapped topic

• G05D1/0246

  using a video camera in combination with image processing means · CPC title

• G06T5/006

  Physics · mapped topic

• G06T5/20

  using local operators · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title