Virtual production based on display assembly pose and pose error correction

Patent metadata

Field	Value
Publication number	US-12020443-B2
Application number	US-202318220227-A
Country	US
Kind code	B2
Filing date	Jul 10, 2023
Priority date	Jul 18, 2022
Publication date	Jun 25, 2024
Grant date	Jun 25, 2024

Abstract

Systems, apparatus, and methods for rendering content based on display assembly pose are disclosed. In an example, motion capture data of a display of a plurality of displays is received, the display moving from a first physical pose to a second physical pose. The motion capture data is processed to determine the coordinates of the second physical pose. A transformation of the second physical pose of the display to a virtual pose of the display is generated. A virtual model of the plurality of displays is updated, the virtual model comprising the virtual pose of the display. The content is rendered on the updated virtual model.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving, by at least one processor, motion capture data of a display of a plurality of displays, the display moving from a first physical pose to a second physical pose;
processing, by the at least one processor, the motion capture data to determine coordinates of the second physical pose;
generating, by the at least one processor, a transformation of the second physical pose of the display to a virtual pose of the display;
updating, by the at least one processor, a virtual model of the plurality of displays, the virtual model comprising the virtual pose of the display; and
rendering, by the at least one processor, content on the display based on the updated virtual model.

2. The computer-implemented method of claim 1 , wherein the computer-implemented method further comprises:
receiving an input to initialize a motion capturing system to generate the motion capture data, the motion capturing system comprising a motion capture device; and
initializing the motion capturing system based on the input, wherein initializing the motion capturing system comprises determining a location and orientation of the motion capture device.

3. The computer-implemented method of claim 2 , wherein the input is a user-based input, and wherein the computer-implemented method further comprises:
projecting a virtual marker on the display based on receiving the user-based input;
determining a presentation pose of the virtual marker, the presentation pose corresponding to the second physical pose; and
determining the coordinates of the second physical pose based on the presentation pose.

4. The computer-implemented method of claim 2 , wherein the input is a sensor based input, and wherein the computer-implemented method further comprises:
receiving streaming data from a sensor configured to collect data associated with the display;
detecting a first change point and a second change point from the streaming data;
determining that the display is in the second physical pose based on detecting the second change point; and
processing the motion capture data to determine the coordinates of the second physical pose based on the determination that the display is in the second physical pose.

5. The computer-implemented method of claim 4 , wherein the first change point and the second change point are detected using a forgetting factor-based change point detection algorithm.

6. The computer-implemented method of claim 4 , wherein determining the coordinates of the second physical pose comprises using a perspective-n-point (PnP) pose computation algorithm.

7. The computer-implemented method of claim 4 , wherein determining the coordinates of the second physical pose comprises determining the coordinates in a coordinate system of a motion capture device used to capture the motion capture data.

8. The computer-implemented method of claim 4 , wherein the computer-implemented method further comprises:
starting reception the motion capture data based on detecting the first change point; and
while the display is moving:
continuing to receive the motion capture data;
processing the motion capture data continuously to determine current coordinates of the display;
generating a current transformation using the current coordinates of the display; and
updating the virtual model of the plurality of displays comprising a current virtual pose of the display, the current virtual pose being associated with the current coordinates of the display.

9. A system comprising:
one or more processors; and
one or more memory storing instructions that, upon execution by the one or more processors, configure the system to:
receive motion capture data of a display of a plurality of displays moving from a first physical pose to a second physical pose;
process the motion capture data to determine coordinates of the second physical pose;
generate a transformation of the second physical pose of the display to a virtual pose of the display;
update a virtual model of the plurality of displays comprising the virtual pose of the display; and
render content on the display based on the updated virtual model.

10. The system of claim 9 , wherein the instructions, upon execution by the one or more processors, further configure the system to:
receive an input to initialize a motion capturing system to capture the motion capture data, the motion capturing system comprising a motion capture device; and
initialize the motion capturing system based on the input, wherein initializing the motion capturing system comprises determining a location and orientation of the motion capture device.

11. The system of claim 10 , wherein the input is a user-based input, and wherein the instructions, upon execution by the one or more processors, further configure the system to:
project a virtual marker on the display based on receiving the user-based input;
determine a presentation pose of the virtual marker, the presentation pose corresponding to the second physical pose; and
determine the coordinates of the second physical pose based on the presentation pose.

12. The system of claim 11 , wherein the motion capturing system comprises at least one motion capturing device, and wherein the virtual marker comprises a plurality of virtual points projected on the display, the at least one motion capturing device configured to capture motion capture data based on tracking the plurality of virtual points.

13. The system of claim 11 , wherein the motion capturing system comprises a plurality of motion capturing devices, and wherein the virtual marker comprises at least one virtual point, the plurality of motion capturing devices configured to capture motion capture data based on tracking the at least one virtual point.

14. The system of claim 11 , wherein the virtual marker covers a portion of the display, and wherein a remaining portion of the display projects a portion of a scene.

15. The system of claim 11 , wherein the virtual marker covers an entirety of the display.

16. The system of claim 11 , wherein the instructions, upon execution by the one or more processors, further configure the system to:
start to receive the motion capture data based on detecting a first change point; and
while the display is moving:
continue to receive the motion capture data;
process the motion capture data continuously to determine current coordinates of the display;
generate a current transformation using the current coordinates of the display; and
generate a current virtual model of the plurality of displays comprising a current virtual pose of the display, the current virtual pose being associated with the current coordinates of the display.

17. One or more non-transitory computer-readable storage media storing instructions that, upon execution on a system, cause the system to perform operations comprising:
receiving motion capture data of a display of a plurality of displays moving from a first physical pose to a second physical pose;
processing the motion capture data to determine coordinates of the second physical pose;
generating a transformation of the second physical pose of the display to a virtual pose of the display;
updating a virtual model of the plurality of displays comprising the virtual pose of the display; and
rendering content on the display based on the updated virtual model.

18. The one or more non-transitory computer-readable storage media of claim 17 , wherein the operations further comprise:
receiving an input to init

Assignees

• Nant Holdings Ip Llc
• Nantstudios Llc

Inventors

• Beziaeva Liudmila A
• Marshall Gary
• Sanchez Adolfo
• Delgado Juan Alfredo Nader
• Demeo Anthony
• Mcspadden Jennifer
• Keeling Shane

Classifications

• G06T2215/16

  Using real world measurements to influence rendering · CPC title

• G06T2207/30204

  Marker · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title