Determining a camera control point for virtual production

Patent metadata

Field	Value
Publication number	US-2025175715-A1
Application number	US-202519036587-A
Country	US
Kind code	A1
Filing date	Jan 24, 2025
Priority date	Jul 18, 2022
Publication date	May 29, 2025
Grant date	—

Abstract

A method including initializing a camera model for a camera device that captures images of a scene, the camera model including extrinsic parameters, intrinsic parameters, and a distortion model. The method further including receiving motion capture data over time indicating current poses of the camera device. The method further including receiving current image data from the camera device. The method further including updating at least one of (i) a parameter of the camera model or (ii) the distortion model, based on the motion capture data and the current image data. The method further including rendering content for display in the virtual production environment based on the update to the camera model.

First claim

What is claimed is:

1 . A system for real-time camera model updating in a virtual production environment, comprising:
a camera device able to capture images of a scene; a motion capture system tracking poses of the camera device; at least one computer readable non-transitory memory storing software instructions; and at least one processor coupled with camera device, motion capture system, and the at least one memory, the processor performing the following operations upon execution of the software instructions:
initializing a camera model for the camera device, the camera model including extrinsic parameters, intrinsic parameters, and a distortion model;
receiving motion capture data over time indicating current poses of the camera device;
receiving current image data from the camera device;
updating at least one of (i) a parameter of the camera model or (ii) the distortion model, based on the motion capture data and the current image data; and
rendering content for display in the virtual production environment based on the update to the camera model.

2 . The system of claim 1 , wherein updating the parameter of the camera model comprises reducing errors between the motion capture data and the current image data by applying a fitting model to the camera model.

3 . The system of claim 2 , wherein the fitting model comprises a Levenberg-Marquardt nonlinear least squares algorithm.

4 . The system of claim 2 , wherein updating the parameter of the camera model further includes:
selecting a subset of the current image data based on reprojection errors; and applying the fitting model to the subset of the current image data to refine the camera model.

5 . The system of claim 1 , wherein updating the distortion model comprises adjusting at least one of: radial distortion coefficients or tangential distortion coefficients.

6 . The system of claim 1 , wherein updating the parameter of the camera model comprises updating the intrinsic parameters.

7 . The system of claim 6 , wherein updating the intrinsic parameters comprises adjusting a focal length of the camera device.

8 . The system of claim 6 , wherein updating the intrinsic parameters comprises adjusting a principal point of the camera device.

9 . The system of claim 1 , wherein updating the parameter of the camera model comprises updating the extrinsic parameters.

10 . The system of claim 1 , further comprising operations of:
predicting a pose of the camera device based on the motion capture data; and pre-computing a transformation for content rendering based on the predicted pose.

11 . The system of claim 1 , wherein the camera device comprises multiple lenses.

12 . The system of claim 1 , wherein the camera device comprises a camera array of multiple individual cameras.

13 . The system of claim 12 , wherein the operations further comprise determining a geometric center of control points of the multiple individual cameras.

14 . The system of claim 1 , wherein the operations further comprise updating the camera model by reducing project errors.

15 . The system of claim 1 , wherein the operations further comprise determining an offset between a marker attached to the camera device and a control point of the camera device based on the update to the camera model.

16 . The system of claim 15 , further comprising the operations of:
setting a pose of a virtual camera in a virtual space based on the determined offset; and wherein rendering the content is based on the pose of the virtual camera.

17 . The system of claim 1 , wherein the operation of rendering content further comprises:
determining a virtual display area based on a current pose of the camera device; determining a portion of the content associated with the virtual display area; and distorting the portion of the content based on the updated distortion model.

18 . The system of claim 1 , wherein the operations further comprise:
tracking poses of one or more objects in the scene using the motion capture system; and updating the rendering of the content based on both the updated camera model and the tracked poses of the one or more objects.

19 . A method for real-time camera model updating in a virtual production environment, comprising:
initializing a camera model for a camera device that captures images of a scene, the camera model including extrinsic parameters, intrinsic parameters, and a distortion model; receiving motion capture data over time indicating current poses of the camera device; receiving current image data from the camera device; updating at least one of (i) a parameter of the camera model or (ii) the distortion model, based on the motion capture data and the current image data; and rendering content for display in the virtual production environment based on the update to the camera model.

20 . One or more non-transitory computer-readable storage media storing instructions that, upon execution by one or more processors of a system, cause the system to perform operations comprising:
initializing a camera model for a camera device that captures images of a scene, the camera model including extrinsic parameters, intrinsic parameters, and a distortion model; receiving motion capture data over time indicating current poses of the camera device; receiving current image data from the camera device; updating at least one of (i) a parameter of the camera model or (ii) the distortion model, based on the motion capture data and the current image data; and rendering content for display in the virtual production environment based on the update to the camera model.

Assignees

• Nant Holdings Ip Llc
• Nantstudios Llc

Inventors

• Beziaeva Liudmila A
• Marshall Gary
• Sanchez Adolfo
• Delgado Juan Alfredo Nader

Classifications

• G06T2207/30244

  Camera pose · CPC title

• G06T2207/30204

  Marker · CPC title

• H04N23/90 Primary

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N17/002

  for television cameras · CPC title

• G06T7/80

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title