Video synthesis system, video synthesis device, and video synthesis method

Patent metadata

Field	Value
Publication number	US-2017096106-A1
Application number	US-201515317001-A
Country	US
Kind code	A1
Filing date	Jun 15, 2015
Priority date	Jun 20, 2014
Publication date	Apr 6, 2017
Grant date	—

Abstract

A video display system performs a video conversion process on a video of a camera mounted on a vehicle and displays a resulting video, and includes a plurality of cameras, a detecting unit that detects an object of interest around the vehicle based on information or the like acquired through the plurality of cameras, other sensors, or a network, a transforming/synthesizing unit that transforms and synthesizes the videos photographed by the plurality of cameras using a shape of a virtual projection plane, a virtual viewpoint, and a synthesis method which are decided according to position information of the object of interest detected by the detecting unit, and a display unit that displays the video that is transformed and synthesized by the transforming/synthesizing unit.

First claim

1 . (canceled)

2 . A video synthesis device that transforms and synthesizes a plurality of videos input from a plurality of cameras and outputs a synthesized video to a video display unit, comprising:
an object-of-interest information acquiring unit that acquires position information related to an object of interest; and a transforming/synthesizing unit that transforms and synthesizes the plurality of videos photographed by the plurality of cameras, wherein the transforming/synthesizing unit transforms and synthesizes a virtual projection state in which the videos photographed by the plurality of cameras are projected onto a virtual projection plane into a video viewed from a virtual viewpoint set according to position information related to the object of interest, and the transforming/synthesizing unit sets the virtual viewpoint at a position at which an angle formed by a straight line connecting the virtual viewpoint with the object of interest and a straight line connecting the object of interest with a camera that images the object of interest among the plurality of cameras is smaller than a predetermined angle.

3 . The video synthesis device according to claim 2 ,
wherein the predetermined angle is decided so that distortion of a video of the object of interest in the synthesized image is minimized according to a distance between the object of interest and the camera that images the object of interest and a deviation amount between a projection position at which the object of interest is projected onto the virtual projection plane and a position of the object of interest.

4 . (canceled)

5 . The video synthesis device according to claim 2 ,
wherein the virtual projection plane is configured with a plane that has the same height as a ground surface and is parallel to the ground surface and a spherical surface arranged at a position farther than the object of interest, and the transforming/synthesizing unit sets at least one of a center position of the spherical surface and a radius of the spherical surface so that the spherical surface of the virtual projection plane is arranged within a predetermined distance from a position of the object of interest when viewed from the camera that photographs the object of interest.

6 . The video synthesis device according to claim 2 ,
wherein the transforming/synthesizing unit synthesizes the videos by blending videos obtained by transforming a virtual projection state in which the plurality of videos are switched at a blending position decided according to the position information related to the object of interest and projected onto the virtual projection plane into the videos viewed from the virtual viewpoint.

7 . The video synthesis device according to claim 6 ,
wherein the transforming/synthesizing unit synthesizes,
when the object of interest exists in a region that is duplicatedly photographed by two neighboring cameras, the videos by blending videos obtained by setting the blending position so that the object of interest does not overlap the blending position and transforming the videos photographed by the two cameras.

8 . The video synthesis device according to claim 2 ,
wherein the object of interest is an object that is inferred based on information related to a speed of the vehicle with which the video synthesis device is equipped, information related to a steering angle, and information related to a traveling direction and exists in a direction having a possibility of a collision with the vehicle.

9 . (canceled)

10 . A video synthesis method of transforming and synthesizing videos input from a plurality of cameras and outputting a synthesized video to a video display unit, comprising:
an object-of-interest information acquisition step of acquiring position information related to an object of interest; and a transformation/synthesis step of transforming and synthesizing the plurality of videos photographed by the plurality of cameras, wherein in the transformation/synthesis step, a virtual projection state in which the videos photographed by the plurality of cameras are projected onto a virtual projection plane is transformed and synthesized into a video viewed from a virtual viewpoint set according to the position information related to the object of interest, and in the transformation/synthesis step, the virtual viewpoint is set at a position at which an angle formed by a straight line connecting the virtual viewpoint with the object of interest and a straight line connecting the object of interest with a camera that images the object of interest among the plurality of cameras is smaller than a predetermined angle.

11 . The video synthesis method according to claim 10 ,
wherein the predetermined angle is decided so that distortion of a video of the object of interest in the synthesized image is minimized according to a distance between the object of interest and the camera that images the object of interest and a deviation amount between a projection position at which the object of interest is projected onto the virtual projection plane and a position of the object of interest.

12 . (canceled)

13 . The video synthesis method according to claim 10 ,
wherein the virtual projection plane is configured with a plane that has the same height as a ground surface and is parallel to the ground surface and a spherical surface arranged at a position farther than the object of interest, and in the transformation/synthesis step, at least one of a center position of the spherical surface and a radius of the spherical surface is set so that the spherical of the virtual projection plane is arranged within a predetermined distance from a position of the object of interest when viewed from the camera that photographs the object of interest.

14 . The video synthesis method according to claim 10 ,
wherein in the transformation/synthesis step, the videos are synthesized by blending videos obtained by transforming a virtual projection state in which the plurality of videos are switched at a blending position decided according to the position information related to the object of interest and projected onto the virtual projection plane into the videos viewed from the virtual viewpoint.

15 . (canceled)

Assignees

• Clarion Co Ltd

Inventors

• Higuchi Haruhiko
• Takanohashi Kenta
• Nakajima Mitsuo
• Sakaniwa Hidenori
• Uchida Yoshitaka
• Nakamura Hiroyuki
• Onozaki Katsuo
• Shioya Takayuki

Classifications

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N23/63

  by using electronic viewfinders · CPC title

• G06V10/247

  by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids · CPC title

• B60R2300/205

  using a head-up display · CPC title

• H04N13/243

  using three or more two-dimensional [2D] image sensors · CPC title