Determining a camera control point for virtual production

Patent metadata

Field	Value
Publication number	US-12244940-B2
Application number	US-202418414192-A
Country	US
Kind code	B2
Filing date	Jan 16, 2024
Priority date	Jul 18, 2022
Publication date	Mar 4, 2025
Grant date	Mar 4, 2025

Abstract

Systems, apparatus, and methods of rendering content based on a control point of a camera device are disclosed. In an example, a marker is attached to the camera device and its pose is tracked over time. Based on a camera model of the camera device, an offset between the marker and the control point is determined. The tracked pose of the marker can be translated and/or rotated according to the offset to estimate a pose of the control point. The rendering of the content is adjusted over time based on the estimated poses of the control point. Upon presentation of the content in a real-world space (e.g., on a display assembly located therein), the camera device can capture the content along with other objects of the real-world space and generate a video stream thereof.

First claim

What is claimed is:

1. A computer-implemented method comprising:
determining a first pose of a marker in a real-world space, wherein the marker is attached to a camera device;
determining an offset between the marker and a control point of the camera device based at least on the first pose;
setting a second pose of a virtual camera in a virtual space based at least on the offset, wherein the virtual camera represents the camera device in the virtual space; and
rendering, on a display, content based at least on the second pose of the virtual camera.

2. The computer-implemented method of claim 1 , wherein determining the offset comprises:
determining a first rotation of the marker in the real-world space based at least on the first pose;
determining a second rotation of the control point based at least on at least one calibration parameter; and
determining a rotation offset based at least on the first rotation and the second rotation.

3. The computer-implemented method of claim 2 , wherein determining the offset further comprises:
determining a first position of the marker in the real-world space based at least on the first pose;
determining a second position of the control point based at least on the at least one calibration parameter; and
determining a position offset based at least on the first rotation, the first position, and the second position.

4. The computer-implemented method of claim 1 , wherein the second pose is set as a translation and rotation of the first pose based at least on the offset.

5. The computer-implemented method of claim 1 , wherein rendering the content comprises:
determining a virtual display area in the virtual space based at least on the second pose;
determining a portion of the content that is associated with the virtual display area;
determining a distortion model based at least on a camera model of the camera device; and
distorting the portion of the content based at least on the distortion model.

6. The computer-implemented method of claim 1 , wherein rendering the content further comprises determining a camera model of the camera device, wherein the camera model is determined based at least on a calibration procedure that comprises:
determining a plurality of poses of an object in the real-world space;
receiving a plurality of images generated by the camera device, wherein the plurality of images represents a video stream that shows the object in the plurality of poses; and
determining, based at least on the plurality of poses and the plurality of images, the camera model of the camera device, wherein the camera model comprises extrinsic camera parameters, intrinsic camera parameters, and a distortion model.

7. The computer-implemented method of claim 1 , further comprising determining at least one calibration parameter of the camera device based on a camera calibration procedure that comprises:
determining a third pose of an object in the real-world space;
receiving an image generated by the camera device, wherein the image shows the object while the object is in the third pose; and
determining, based at least on the third pose and the image, the at least one calibration parameter.

8. The computer-implemented method of claim 7 , wherein the camera calibration procedure is performed while one of the camera device or the object is static and a remaining one of the camera device or the object is movable.

9. The computer-implemented method of claim 7 , wherein the camera calibration procedure further comprises:
determining a plurality of poses of the object in the real-world space while the camera device remains in a same pose;
receiving a plurality of images generated by the camera device, wherein the plurality of images represents a video stream that shows the object in the plurality of poses;
determining that the third pose is shown in the image;
determining a projection error between the image and a projection of the object in the third pose onto an image plane of a camera model of the camera device, wherein the camera model comprises the at least one calibration parameter;
selecting a subset of images of the plurality of images based at least on projection errors, wherein the subset of images is associated with a subset of poses of the plurality of poses; and
updating the camera model based at least on the subset of images and the subset of poses.

10. The computer-implemented method of claim 7 , wherein the camera calibration procedure further comprises:
determining a projection error between the image and a projection of the object in the third pose onto an image plane of a camera model of the camera device, wherein the camera model comprises the at least one calibration parameter; and
updating the camera model based at least on a fitting model that reduces projection errors, wherein an output of the fitting model indicates one or more parameters of the camera model.

11. The computer-implemented method of claim 7 , wherein the camera calibration procedure further comprises:
determining a first estimated pose of the camera device based at least on the third pose of the object and the image;
determining a projection error between the image and a projection of the object in the third pose onto an image plane of a camera model of the camera device, wherein the camera model comprises the at least one calibration parameter;
associating a first weight with the first estimated pose based at least on the projection error;
determining a final estimated pose of the camera device based at least on the first estimated pose and the first weight; and
updating the camera model based at least on a fitting model that uses the final estimated pose and projection errors.

12. The computer-implemented method of claim 1 , further comprising determining at least one calibration parameter of the camera device based at least on a camera model of the camera device, wherein the camera model is determined based at least on a camera calibration procedure that comprises:
receiving motion capture data that represents a plurality of poses of an object in the real-world space over time while the camera device remains in a same pose;
determining, based at least on a change in the motion capture data, a first timing of a third pose of the object; and
determining, based at least on the third pose, the at least one calibration parameter.

13. The computer-implemented method of claim 12 , wherein the camera calibration procedure further comprises:
receiving image data that represents a plurality of images generated by the camera device over time while the camera device remains in the same pose, wherein the plurality of images represents a video stream that shows the object in the plurality of poses; and
associating the third pose with an image of the plurality of images based at least on the timing of the third pose and a second timing of the image, wherein the at least one calibration parameter is determined further based at least on the timing.

14. A system comprising:
one or more processors; and
one or more memory storing instructions that, upon execution by the one or more processors, configure the system to:
determine a first pose of a marker in a real-world space, wherein the marker is attached to a camera device;
determine an offset between the marker and a control point of the camera device based at least on the first pose;
set a second pose of a virtual camera in a virtual space based at least on the offset, wherein the virtual camera represents the camera device in the virtual space; and
render, on a display, content based at least on the second pose of the virtual

Assignees

• Nant Holdings Ip Llc
• Nantstudios Llc

Inventors

• Beziaeva Liudmila A
• Marshall Gary
• Sanchez Adolfo
• Delgado Juan Alfredo Nader

Classifications

• G06T2207/30244

  Camera pose · CPC title

• G06T2207/30204

  Marker · CPC title

• H04N23/90 Primary

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N17/002

  for television cameras · CPC title

• G06T7/80

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title