Machine surround view system and method for generating 3-dimensional composite surround view using same

Patent metadata

Field	Value
Publication number	US-10475154-B2
Application number	US-201715675224-A
Country	US
Kind code	B2
Filing date	Aug 11, 2017
Priority date	Aug 11, 2017
Publication date	Nov 12, 2019
Grant date	Nov 12, 2019

Abstract

A surround view system for a machine is provided. The surround view system includes a plurality of image capturing devices generating image data of surroundings of the machine, and an object detection system for detecting an object in a target field of view of the machine and generating object position data corresponding to the object. The surround view system also includes an image processing system configured to generate an initial 3-dimensional composite surround view by projecting the image data on a virtual model corresponding to the machine. The virtual model has a 3-dimensional shape based on an initial calibration position data. The image processing system is also configured to modify the 3-dimensional shape of the virtual model based on the object position data, and generate an updated 3-dimensional composite surround view by projecting the image data on the virtual model having a modified 3-dimensional shape.

First claim

What is claimed is:

1. A surround view system for a machine, the surround view system including:
one or more devices configured to generate image data of surroundings of the machine;
one or more processors configured to:
generate an initial 3-dimensional composite surround view by projecting the image data on a virtual model corresponding to the machine,
wherein the virtual model has a 3-dimensional shape based on initial calibration position data;

detect a first object and a second object in a target field of view of the machine;
filter, based on a size of the first object and a size of the second object, data for the first object and the second object to determine to consider the first object and ignore the second object;
modify, to obtain a modified 3-dimensional shape, the 3-dimensional shape of the virtual model based on a position of the first object after filtering data for the first object and the second object; and
generate an updated 3-dimensional composite surround view by projecting the image data on the virtual model having the modified 3-dimensional shape.

2. The surround view system of claim 1 , wherein, when modifying the 3-dimensional shape of the virtual model, the one or more processors are configured to:
modify an inner diameter of the virtual model and an outer diameter of the virtual model independently.

3. The surround view system of claim 1 , wherein, when modifying the 3-dimensional shape of the virtual model, the one or more processors are configured to:
modify an inner diameter of the virtual model, and
modify an outer diameter of the virtual model as a function of the inner diameter.

4. The surround view system of claim 1 ,
wherein the virtual model is a bowl-shaped model, and
wherein the modified 3-dimensional shape is asymmetrical.

5. The surround view system of claim 1 , wherein at least one of a radar, a laser, LiDAR, video, or ultrasound is used to detect the first object.

6. The surround view system of claim 5 , wherein the one or more processors are configured to:
generate object position data corresponding to the position of the first object.

7. The surround view system of claim 1 , further including:
a user display configured to display the updated 3-dimensional composite surround view.

8. A method including:
generating image data of surroundings of a machine;
generating an initial 3-dimensional composite surround view by projecting the image data on a virtual model,
wherein the virtual model has a 3-dimensional shape based on an initial calibration position data;

detecting a first object and a second object in a target field of view of the machine;
filtering, based on a size of the first object and a size of the second object, data for the first object and the second object to determine to consider the first object and ignore the second object;
modifying, to obtain a modified 3-dimensional shape, the 3-dimensional shape of the virtual model based on object position data, corresponding to the first object, after filtering the data for the first object and the second object;
generating an updated 3-dimensional composite surround view by projecting the image data on the virtual model having a modified 3-dimensional shape; and
providing, for display, the updated 3-dimensional composite surround view on a user display.

9. The method of claim 8 , wherein modifying the 3-dimensional shape includes:
modifying an inner diameter of the virtual model and an outer diameter of the virtual model independently.

10. The method of claim 8 , wherein modifying the 3-dimensional shape includes:
modifying an inner diameter of the virtual model, and
modifying an outer diameter of the virtual model as a function of the inner diameter.

11. The method of claim 8 , wherein modifying the 3-dimensional shape of the virtual model includes:
generating an asymmetrical shape of the virtual model.

12. The method of claim 8 , wherein the updated 3-dimensional composite surround view is generated and displayed while the machine is in motion.

13. The method of claim 8 , further including:
generating the object position data.

14. An image processing system for generating a 3-dimensional composite surround view, the image processing system including:
a memory; and
a processor configured to:
receive image data of surroundings of a machine;
generate an initial 3-dimensional composite surround view by projecting the image data on a virtual model;
receive object position data corresponding to a first object and a second object in a target field of view of the machine;
filter, based on a size of the first object and a size of a second object, the object position data corresponding to the first object and the second object to determine to consider the first object and ignore the second object;
modify, to obtain a modified 3-dimensional shape, the 3-dimensional shape of the virtual model based on a position of the first object after filtering the object position data corresponding to the first object and the second object; and
generate an updated 3-dimensional composite surround view by projecting the image data on the virtual model having the modified 3-dimensional shape.

15. The image processing system of claim 14 , wherein, when modifying the 3-dimensional shape of the virtual model, the processor is configured to:
modify an inner diameter of the virtual model and an outer diameter of the virtual model independently.

16. The image processing system of claim 14 , wherein, when modifying the 3-dimensional shape of the virtual model, the processor is configured to:
modify an inner diameter of the virtual model, and
modify an outer diameter of the virtual model as a function of the inner diameter.

17. The image processing system of claim 14 , wherein the modified 3-dimensional shape is asymmetrical.

18. The image processing system of claim 14 , further including:
at least four cameras configured to generate the image data.

19. The image processing system of claim 18 , wherein each of the four cameras includes a fisheye lens.

20. The image processing system of claim 14 , wherein the processor is further configured to:
transmit the updated 3-dimensional composite surround view to a user display.

Assignees

• Caterpillar Inc

Inventors

• Husted Douglas Jay
• Zwetz Ii David Lee
• Roesner Adam Paul
• Lin Edward Yii-Hsin
• Ali Hassen
• Petrany Peter

Classifications

• B60R2300/105

  using multiple cameras · CPC title

• B60R2300/102

  using 360 degree surveillance camera system · CPC title

• B60R2300/303

  using joined images, e.g. multiple camera images · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T3/4038 Primary

  Image mosaicing, e.g. composing plane images from plane sub-images · CPC title