System and method for fisheye image processing

Patent metadata

Field	Value
Publication number	US-12190465-B2
Application number	US-202418437734-A
Country	US
Kind code	B2
Filing date	Feb 9, 2024
Priority date	Oct 19, 2018
Publication date	Jan 7, 2025
Grant date	Jan 7, 2025

Abstract

A system and method for fisheye image processing can be configured to: receive fisheye image data from at least one fisheye lens camera associated with an autonomous vehicle, the fisheye image data representing at least one fisheye image frame; partition the fisheye image frame into a plurality of image portions representing portions of the fisheye image frame; warp each of the plurality of image portions to map an arc of a camera projected view into a line corresponding to a mapped target view, the mapped target view being generally orthogonal to a line between a camera center and a center of the arc of the camera projected view; combine the plurality of warped image portions to form a combined resulting fisheye image data set representing recovered or distortion-reduced fisheye image data corresponding to the fisheye image frame; generate auto-calibration data representing a correspondence between pixels in the at least one fisheye image frame and corresponding pixels in the combined resulting fisheye image data set; and provide the combined resulting fisheye image data set as an output for other autonomous vehicle subsystems.

First claim

What is claimed is:

1. A system comprising:
a data processor; and
a fisheye image processing system, executable by the data processor, the fisheye image processing system comprising:
a fisheye image processing module configured to,
receive fisheye image data from at least one fisheye lens camera associated with an autonomous vehicle, the fisheye image data representing at least one fisheye image frame;

an image warping module configured to,
partition the fisheye image data into a plurality of image portions representing portions of the at least one fisheye image frame; and
warp each of the plurality of image portions to map an arc of a camera projected view into a line corresponding to a mapped target view, the mapped target view being generally orthogonal to a line between a camera center and a center of the arc of the camera projected view;

an image stitching module configured to,
combine the plurality of warped image portions to form a combined resulting image representing recovered or distortion-reduced image data from the at least one fisheye lens camera; and

an object extraction module configured to:
perform object extraction on the combined resulting image to identify extracted objects from the combined resulting image,
wherein the extracted objects are used to determine one or more control operations for the autonomous vehicle,

wherein the fisheye image processing module is further configured to,
perform an optimization operation of the fisheye image data, wherein the optimization operation comprises determining a pixel-to-pixel correspondence between the fisheye image data and the combined resulting image by storing a pixel position in the combined resulting image using a mapping object.

2. The system of claim 1 being further configured to align the mapped target view with a line parallel to a side of the autonomous vehicle.

3. The system of claim 1 , wherein the fisheye image processing system further comprises an offline processing stage and an online processing stage, wherein the offline processing stage is used to pre-process the fisheye image data to generate one or more combined image transformations.

4. The system of claim 1 wherein the at least one fisheye lens camera is from a group consisting of: a left side fisheye lens camera, a right side fisheye lens camera, and a top mounted fisheye lens camera.

5. The system of claim 1 being further configured to obtain parameters of the fisheye lens, the parameters including a fisheye lens radius, a lens aperture, a focal length, and a target field of view angle.

6. The system of claim 1 being further configured to map pixels in the fisheye image frame to pixels in the combined resulting fisheye image and to combine all of the pixel mappings to produce an end-to-end mapping for optimization.

7. The system of claim 1 , wherein the fisheye image processing module is implemented using an execution environment with enhanced data security.

8. The system of claim 1 , wherein an efficiency of the fisheye image processing module is enhanced by use of a Fast Fourier Transform function.

9. The system of claim 1 being further configured to reduce redundant computation time for each image, so that the system reaches a real-time performance of 50 Hz per image.

10. A method comprising:
receiving, by a fisheye image processing module, fisheye image data from at least one fisheye lens camera associated with an autonomous vehicle, the fisheye image data representing at least one fisheye image frame;
partitioning the fisheye image data into a plurality of image portions representing portions of the at least one fisheye image frame;
warping, by an image warping module, each of the plurality of image portions to map an arc of a camera projected view into a line corresponding to a mapped target view, the mapped target view being generally orthogonal to a line between a camera center and a center of the arc of the camera projected view;
combining, by an image stitching module, the plurality of warped image portions to form a combined resulting image representing recovered or distortion-reduced image data from the at least one fisheye lens camera; and
performing, by an object extraction module, object extraction on the combined resulting image to identify extracted objects from the combined resulting image,
wherein the extracted objects are used to determine one or more control operations for the autonomous vehicle,
wherein the fisheye image processing module is configured to,
optimize the fisheye image data by determining a pixel-to-pixel correspondence between the fisheye image data and the combined resulting image by storing a pixel position in the combined resulting image using a mapping object.

11. The method of claim 10 including aligning the mapped target view with a line parallel to a side of the autonomous vehicle.

12. The method of claim 10 wherein the at least one fisheye lens camera is from a group consisting of: a left side fisheye lens camera, a right side fisheye lens camera, and a top mounted fisheye lens camera.

13. The method of claim 10 including obtaining parameters of the fisheye lens, the parameters including a fisheye lens radius, a lens aperture, a focal length, and a target field of view angle.

14. The method of claim 10 including mapping pixels in the fisheye image frame to pixels in the combined resulting fisheye image and combining all of the pixel mappings to produce an end-to-end mapping for optimization.

15. The method of claim 10 , wherein the fisheye image processing module is implemented using an execution environment with enhanced data security.

16. The method of claim 10 , wherein an efficiency of the fisheye image processing module is enhanced by use of a Fast Fourier Transform function.

17. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to
receive, by a fisheye image processing module, fisheye image data from at least one fisheye lens camera associated with an autonomous vehicle, the fisheye image data representing at least one fisheye image frame;
partition the fisheye image data into a plurality of image portions representing portions of the at least one fisheye image frame;
warp, by an image warping module, each of the plurality of image portions to map an arc of a camera projected view into a line corresponding to a mapped target view, the mapped target view being generally orthogonal to a line between a camera center and a center of the arc of the camera projected view;
combine, by an image stitching module, the plurality of warped image portions to form a combined resulting image representing recovered or distortion-reduced image data from the at least one fisheye lens camera; and
perform, by an object extraction module, object extraction on the combined resulting image to identify extracted objects from the combined resulting image,
wherein the extracted objects are used to determine one or more control operations for the autonomous vehicle,

wherein the fisheye image processing module is configured to,
optimize the fisheye image data by determining a pixel-to-pixel correspondence between the fisheye image data and the combined resulting image by storing a pixel position in the combined resulting image using a mapping object.

18. The non-transitory machine-useable storage medium of claim 17 being further configured to align the mapped target view with a line parallel to a side of the autonomous vehicle.

19. The non-transitory machine

Assignees

• Tusimple Inc

Inventors

• Yan Zhipeng
• Chen Pengfei
• Wang Panqu

Classifications

• G05D1/249

  from positioning sensors located off-board the vehicle, e.g. from cameras · CPC title

• G06T5/80

  Geometric correction · CPC title

• G06T5/20

  using local operators · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title

• G05D1/0246

  using a video camera in combination with image processing means · CPC title