Dynamic seam adjustment of image overlap zones from multi-camera source images

Patent metadata

Field	Value
Publication number	US-10992860-B2
Application number	US-201916370620-A
Country	US
Kind code	B2
Filing date	Mar 29, 2019
Priority date	Mar 29, 2019
Publication date	Apr 27, 2021
Grant date	Apr 27, 2021

Abstract

Embodiments of the present disclosure are directed to providing a more accurate presentation of a vehicle's surroundings when the presentation combines images from multiple cameras. Cameras may capture the same portion of the environment. If an object is within view of at least two cameras, each camera produces a portion of an overall view, such as a 360 degree synthetic top-down view. The seam, angle and/or geometry, angle between any two images is dynamically determined, such as due to proximity (or lack thereof) to an object. As a result the synthetic top-down view may present an image of an object more prominently and/or avoid having the seam between camera images fall on the image of the object, which may otherwise result in the image of the object being omitted from both camera images.

First claim

What is claimed is:

1. A method for processing images of a vehicle's surroundings, comprising:
obtaining a first image from a first camera having a first vantage of the vehicle's surroundings;
obtaining a second image from a second camera having a second vantage of the vehicle's surroundings that is different from the first vantage;
determining whether an object is present within a field of view of both the first camera and the second camera;
upon determining that the object is present within the field of view, selecting a portion of the first image and a portion of the second image to include an image of the object in one of the selected portion of the first image or the selected portion of the second image and not both the selected portion of the first image and the selected portion of the second image; and
stitching together the selected portion of the first image and selected portion of the second image into a unified image; and
wherein the selection of the portion of the first image and the portion of the second image is further selected based on a distance of the object and further comprising selecting a seam angle of the portion of the first image in accordance with the distance of the object applied to a sigmoid function.

2. The method of claim 1 , further comprising:
monitoring a relative position between the object and the vehicle; and
in response to a change in the relative position, reselecting the portion of the first image and the portion of the second image to maintain the image of the object in one of the portion of the first image or the portion of the second image and not both.

3. The method of claim 1 , further comprising, presenting the unified image on a display within the vehicle.

4. The method of claim 1 , further comprising converting the unified image into a synthetic reconstructed view of the vehicle's surroundings.

5. The method of claim 1 , wherein the sigmoid function is bounded by a range of the angle of the first image.

6. The method of claim 5 , wherein the range of the seam angle of the portion of the first image is between 30 and 60 degrees and transitions therebetween as object distances transition from substantially 150 cm and closer to substantially 150 cm and farther, respectively.

7. The method of claim 1 , wherein selecting the portion of the first image and the portion of the second image to include the image of the object in one of the portion of the first image or the portion of the second image and not both, further comprises obtaining an outline of the object, making a mask from the outline, and providing, as a first portion of the unified image, an image of an interior of the outline from one of the first image or the second image and providing, as a second portion of the unified image, an image of an exterior of the outline from the other of the first image or second image.

8. The method of claim 1 , wherein the object comprises an irregular feature of a roadway determined to represent a hazard to at least one of the vehicle or the object if at least a portion of the vehicle were to contact the object.

9. The method of claim 1 , wherein the selection of the portion of the first image and the portion of the second image is further selected based on a currently selected direction of travel of the vehicle and comprises selecting an angle of the portion of the first image in accordance with the one of the first image or second image most-facing a direction of travel.

10. A system for processing images of a vehicle's surroundings, comprising:
a first camera having a first vantage of the vehicle's surroundings;
a second camera having a second vantage of the vehicle's surroundings;
a processor receiving image data from each of the first camera and the second camera;
wherein the processor obtains a first image from the first camera;
wherein the processor obtains a second image from the second camera;
wherein the processor determines whether an object is present within a field of view of both the first camera and the second camera;
wherein the processor, upon determining that the object is present within the field of view, selects a portion of the first image and a portion of the second image to include an image of the object in one of the selected portion of the first image or the selected portion of the second image and not both the selected portion of the first image and the selected portion of the second image;
wherein the processor stitches together the selected portion of the first image and selected portion of the second image into a unified image; and
wherein the selection of the portion of the first image and the portion of the second image is further selected based on a distance of the object and further comprising selecting a seam angle of the portion of the first image in accordance with the distance of the object applied to a sigmoid function.

11. The system of claim 10 , further comprising:
a display device within a cabin of the vehicle; and
wherein the processor provides the unified image for display by the display device.

12. The system of claim 11 , wherein the processor converts the unified image into a synthetic reconstructed view of the vehicle's surroundings for display on the display device.

13. The system of claim 10 , further comprising:
a distance measuring component; and
wherein the processor receives distance information to the object from the distance measuring component; and
wherein the processor, in response to a change in a relative distance between the vehicle and the object, reselects the portion of the first image and the portion of the second image to maintain the image of the object in one of the portion of the first image or the portion of the second image and not both.

14. The system of claim 13 , wherein the sigmoid function is bounded by a range of the angle of the portion of the first image and is between 30 and 60 degrees and transitions therebetween as object distances transition from substantially 150 cm and closer to substantially 150 cm and farther, respectively.

15. The system of claim 10 , wherein the processor selects the portion of the first image and the portion of the second image to include the image of the object in one of the portion of the first image or the portion of the second image and not both, further comprises the processor obtaining an outline of the object, making a mask from the outline, and providing, as a first portion of the unified image, an image of an interior of the outline from one of the first image or the second image and providing, as a second portion of the unified image, an image of an exterior of the outline from the other of the first image or second image.

16. The system of claim 10 , wherein the object comprises an irregular feature of a roadway determined to represent a hazard to at least one of the vehicle or the object if at least a portion of the vehicle were to contact the object.

17. A vehicle, comprising:
a power source;
a motor selectively connected to the power source to provide locomotive force to the vehicle;
a cabin to convey therein at least one human;
a first camera having a first vantage of the vehicle's surroundings;
a second camera having a second vantage of the vehicle's surroundings;
a processor receiving image data from each of the first camera and the second camera;
wherein the processor obtains a first image from the first camera;
wherein the processor obtains a second image from the second camera;
wherein the processor determines whether an image of an object is present within a field of view of both

Assignees

• Nio Usa Inc

Inventors

• Liang Anthony Tao
• Elangovan Vidya
• Ramakrishnan Divya
• Jain Prashant

Classifications

• G06V10/803

  of input or preprocessed data · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N7/181 Primary

  for receiving images from a plurality of remote sources · CPC title

• G06F18/251

  of input or preprocessed data · CPC title