Dynamic image blending for multiple-camera vehicle systems

Patent metadata

Field	Value
Publication number	US-11184531-B2
Application number	US-201616325171-A
Country	US
Kind code	B2
Filing date	Oct 12, 2016
Priority date	Dec 21, 2015
Publication date	Nov 23, 2021
Grant date	Nov 23, 2021

Abstract

A method and system for generating a composite video for display in a vehicle. A plurality of video streams are generated from a plurality of video cameras configured to be positioned on the vehicle. The video streams are transformed by an electronic processor to create a virtual camera viewpoint. The transformed video streams are combined to generate a composite video including a portion of a first image that is generated from a first one of the video cameras. The electronic processor detects an object external to the vehicle and determines whether the object at least partially obscures the portion of the first image. When the object at least partially obscures the portion of the first image, the electronic processor supplements the portion of the first image with a portion of a second image that is generated by a second one of the video cameras.

First claim

What is claimed is:

1. A method of generating a composite video for display in a vehicle, the method comprising:
generating a plurality of video streams from a plurality of video cameras configured to be positioned on the vehicle;
transforming one or more of the plurality of video streams to create a virtual camera viewpoint;
combining the plurality of transformed video streams to generate a composite video including a portion of a first image that is generated from a first one of the plurality of video cameras;
detecting, with an electronic processor, an object external to the vehicle;
determining whether the object at least partially obscures the portion of the first image;
supplementing the portion of the first image with a portion of a second image that is generated by a second one of the plurality of video cameras when the object at least partially obscures the portion of the first image to create a supplemented composite video, wherein the supplemented composite video supplemented composite video includes a top-down field of view of the portion of the first image supplemented with the portion of the second image;
classifying the object with the electronic processor;
generating a matching image based on the classification of the object;
overlaying the matching image on the detected object in the composite video as an augmented object, wherein the augmented object is the visual representation of the object;
receiving a feature of the object from at least one sensor;
determining, with the electronic processor, that the object is distorted in the supplemented composite video; and
in response to determining that the object is distorted, adjusting the matching image based on the feature before overlaying the matching image on the object in the composite video as an augmented object.

2. The method of claim 1 , wherein generating the plurality of video streams from the plurality of video cameras includes generating a plurality of wide-angle images with the plurality of video cameras.

3. The method of claim 2 , wherein transforming the one or more of the plurality of video streams includes transforming the plurality of wide-angle images to a plurality of rectilinear images.

4. The method of claim 1 , wherein combining the plurality of transformed video streams to generate the composite video includes stitching together the plurality of transformed video streams such that a composite image is formed from the virtual camera viewpoint.

5. The method of claim 1 , wherein determining whether the object at least partially obstructs the portion of the first image includes determining that a region of a first field of view of the first one of the plurality of video cameras is blocked by the object.

6. The method of claim 5 , further comprising determining whether the second one of the plurality of video cameras has a second field of view that captures at least part of the region blocked by the object.

7. The method of claim 6 , wherein supplementing the portion of the first image with the portion of the second image occurs when the second field of view captures the at least part of the region blocked by the object, wherein the portion of the second image includes the at least part of the region blocked by the object captured by the second field of view.

8. The method of claim 7 , wherein supplementing the portion of the first image with the portion of the second image includes overlaying the portion of the second image onto the portion of the first image within the composite video.

9. A system for generating a composite video to display in a vehicle, the system comprising:
a plurality of video cameras that generate a plurality of video streams, the plurality of video cameras configured to be positioned on the vehicle;
a display; and
an electronic processor communicatively coupled to the plurality of video cameras and the display, the electronic processor configured to
transform the plurality of video streams to create a virtual camera viewpoint;
combine the plurality of transformed video streams to generate a composite video including a portion of a first image that is generated from a first one of the plurality of video cameras;
detect, with an electronic processor, an object external to the vehicle;
determine whether the object at least partially obscures the portion of the first image; and
supplement the portion of the first image with a portion of a second image that is generated by a second one of the plurality of video cameras when the object at least partially obscures the portion of the first image, wherein the supplemented composite video includes a top-down field of view of the portion of the first image supplemented with the portion of the second image
classify the object with the electronic processor;
generate a matching image based on the classification of the object;
overlay the matching image on the detected object in the composite video as an augmented object, wherein the augmented object is the visual representation of the object;
receive a feature of the object from at least one sensor;
determine whether the object is distorted in the supplemented composite video; and
in response to determining that the object is distorted, adjust the matching image based on the feature before overlaying the matching image on the object in the composite video as an augmented object.

10. The system of claim 9 , wherein the electronic processor is configured to transform the plurality of video streams to create the virtual camera viewpoint by transforming a plurality of wide-angle images to a plurality of rectilinear images and stitching together the plurality of transformed video streams such that a composite image is formed from the virtual camera viewpoint.

11. The system of claim 9 , wherein the electronic processor is configured to determine whether the object at least partially obstructs the portion of the first image by determining that a region of a first field of view of the first one of the plurality of video cameras is blocked by the object.

12. The system of claim 11 , wherein the electronic processor is further configured to determine whether the second one of the plurality of video cameras has a second field of view that captures at least part of the region blocked by the object.

13. The system of claim 12 , wherein the electronic processor is configured to supplement the portion of the first image with the portion of the second image when the second field of view captures the at least part of the region blocked by the object, wherein the portion of the second image includes the at least part of the region blocked by the object captured by the second field of view.

14. The system of claim 9 , wherein the electronic processor is configured to overlay the matching image on the object in the composite video by replacing pixels representing the object with a model of the object from a viewpoint that best matches the virtual camera viewpoint.

15. The method of claim 1 , wherein receiving the feature of the object includes receiving a color of the object.

16. The system of claim 9 , wherein the matching image is a top-down image of a virtual vehicle, the virtual vehicle matching the make and model of the object.

Assignees

• Bosch Gmbh Robert

Inventors

• Sypitkowski Greg
• Graf Patrick
• Miller James Stephen

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• H04N23/698 Primary

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• G06T2207/30261

  Obstacle · CPC title

• G06T3/4038

  Image mosaicing, e.g. composing plane images from plane sub-images · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title