3D graphic insertion for live action stereoscopic video

Patent metadata

Field	Value
Publication number	US-9699438-B2
Application number	US-82958010-A
Country	US
Kind code	B2
Filing date	Jul 2, 2010
Priority date	Jul 2, 2010
Publication date	Jul 4, 2017
Grant date	Jul 4, 2017

Abstract

Systems and methods for the insertion of graphics into stereoscopic live action 3D video using image models taking into account positional data from the stereoscopic camera rig.

First claim

What is claimed is:

1. A method of inserting a 3D graphic element into a 3D stereoscopic image generated from two cameras disposed on a camera rig, and compensating for optical effects of the camera rig, comprising:
acquiring a first 2D video image frame captured by a first camera, the first camera being one of the cameras on the camera rig;
acquiring first camera instrumentation data associated with the capture of the first 2D video image frame by the first camera;
acquiring camera rig information associated with the camera rig;
calculating one or more “unnatural disparities” associated with at least one of the first camera and the camera rig, based on the camera rig information including camera toe-in angle and camera separation, and at least one of the first camera instrumentation data and the first 2D video image frame, the “unnatural disparities” comprising at least a keystoning effect;
adjusting the first 2D image frame based on the “unnatural disparities,” and providing a first adjusted 2D image;
converting the 3D graphic element to a first adjusted 2D graphic element using a first virtual camera graphic synthesizer that models imaging of the first camera using a first image formation model, the first image formation model using first camera parameter values derived from at least one of the first camera instrumentation data and the first 2D video image frame; and
incorporating the first adjusted 2D graphic element into the first adjusted 2D image frame to generate a first combined video image frame.

2. The method of claim 1 , wherein the “unnatural disparities” further comprises at least one of radial distortion and tangential distortion.

3. The method of claim 1 , wherein the converting the 3D graphic element to a first adjusted 2D graphic element, comprises adjusting for at least one of radial distortion and tangential distortion, associated with the first camera.

4. The method of claim 1 , wherein the converting the 3D graphic element to a first adjusted 2D graphic element, further comprises adjusting for a first distortion associated with the first camera to the extent that the first adjusted 2D image based on the unnatural disparities did not adjust for the first distortion.

5. The method of claim 4 , wherein the first distortion is at least one of radial distortion and tangential distortion.

6. The method of claim 1 , wherein the first image formation model does not correct for the keystoning effect.

7. The method of claim 1 , further comprising calculating the first camera parameter values using at least one of: first video analysis of the 2D video image frame, and first signal processing of the first camera instrumentation data.

8. The method of claim 1 , wherein the camera rig comprises more than two cameras.

9. The method of claim 1 , further comprising:
acquiring a second 2D video image frame captured by a second camera, the second camera being another one of the cameras on the camera rig;
acquiring second camera instrumentation data associated with the capture of the second 2D video image frame by the camera;
calculating one or more “unnatural disparities” associated with at least one of the second camera and the camera rig, based on the camera rig information including camera toe-in angle and camera separation, and at least one of the second camera instrumentation data and the second 2D video image frame, the “unnatural disparities” comprising at least a keystoning effect;
adjusting the second 2D image frame based on the “unnatural disparities,” and providing a second adjusted 2D image;
converting the 3D graphic element to a second adjusted 2D graphic element using a second virtual camera graphic synthesizer that models imaging of the second camera using a second image formation model, the second image formation model using second camera parameter values derived from at least one of the second camera instrumentation data and the second 2D video image frame; and
incorporating the adjusted 2D graphic element into the adjusted 2D image frame to generate a second combined video image frame.

10. The method of claim 9 , further comprising providing a two channel video signal indicative of the first combined video image frame and the second combined video image frame, and wherein the first combined video image frame and the second combined video image frame are captured simultaneously.

11. The method of claim 9 , further comprising correcting the first combined video image frame and the second combined video image frame for offset parallax based at least on the camera rig information.

12. A system for inserting a 3D graphic element into a 3D stereoscopic image generated from two cameras disposed on a camera rig, the camera rig having camera rig information associated with the camera rig, and compensating for optical effects of the camera rig, comprising:
a first camera, configured to capture a first 2D video image frame, the first camera being one of the cameras on the camera rig, the first camera providing first camera instrumentation data associated with the capture of the first 2D video image frame by the first camera;
a first 2D remapper configured to calculate one or more “unnatural disparities” associated with at least one of the first camera and the camera rig, based on the camera rig information including camera toe-in angle and camera separation, and at least one of the first camera instrumentation data and the first 2D video image frame, the “unnatural disparities” comprising at least a keystoning effect, and configured to adjust the first 2D image frame based on the “unnatural disparities,” the remapper providing a first adjusted 2D image;
a first virtual camera graphic synthesizer configured to convert the 3D graphic element to a first adjusted 2D graphic element, the synthesizer arranged to model imaging of the first camera using a first image formation model, the first image formation model using first camera parameter values derived from at least one of the first camera instrumentation data and the first 2D video image frame; and
a first keyer configured to incorporate the first adjusted 2D graphic element into the first adjusted 2D image frame to generate a first combined video image frame.

13. The system of claim 12 , wherein the “unnatural disparities” further comprises at least one of radial distortion and tangential distortion.

14. The system of claim 12 , wherein the synthesizer is configured to adjust for at least one of radial distortion and tangential distortion, associated with the first camera.

15. The system of claim 12 , wherein the synthesizer is further configured to adjust for a first distortion associated with the first camera to the extent that the first adjusted 2D image based on the unnatural disparities did not adjust for the first distortion.

16. The system of claim 15 , wherein the first distortion is at least one of radial distortion and tangential distortion.

17. The system of claim 12 , wherein the first image formation model does not correct for the keystoning effect.

18. The system of claim 12 , wherein the synthesizer is configured to calculate the first camera parameter values using at least one of: first video analysis of the 2D video image frame, and first signal processing of the first camera instrumentation data.

19. The system of claim 12 , wherein the camera rig comprises more than two cameras.

20. The system of claim 12 , further comprising:
a second camera, configured to capture a second 2D video image frame, the second camera being another o

Assignees

• Walsh Peter
• Disney Entpr Inc

Inventors

• Walsh Peter

Classifications

• H04N13/004

  Electricity · mapped topic

• H04N13/007 Primary

  Electricity · mapped topic

• H04N13/0242

  Electricity · mapped topic

• H04N13/0239

  Electricity · mapped topic

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title