Active marker strobing for performance capture communication

Patent metadata

Field	Value
Publication number	US-11288496-B2
Application number	US-202117333883-A
Country	US
Kind code	B2
Filing date	May 28, 2021
Priority date	Feb 28, 2020
Publication date	Mar 29, 2022
Grant date	Mar 29, 2022

Abstract

The present description relates to light patterns used in a live action scene of a visual production to encode information associated with objects in the scene, such as movement and position of the objects. A data capture system includes active markers that emit light of a particular wavelength in predefined strobing patterns. In some implementations, the active markers are instructed to emit an assigned signature pattern of light through a signal controller sending signals to a control unit. Various components are synchronized such that pulsing of light corresponds to time slices and particular frames captured by the performance capture system. The data representing the pattern is embedded in illuminated and blank frames. Frames showing the light pattern are analyzed to extract information about the active markers, such as identification of the active markers and objects to which they are attached.

First claim

We claim:

1. A method for recognizing active markers in a live action scene for performance capture associated with a virtual production, the method comprising:
synchronizing a signal controller with a rate of capture of one or more sensor devices;
transmitting signals by the signal controller, to instruct one or more active markers to emit electromagnetic radiation according to a pattern including a sequence of electromagnetic radiation pulses and gap periods between the pulses, during specified time slots in which a presence of the electromagnetic radiation coincides with one or more predefined capture frames and an absence of the electromagnetic radiation coincides with one or more predefined blank frames of the one or more sensor devices;
generating data associated with the presence of electromagnetic radiation represented in one or more capture frames and absence of electromagnetic radiation represented in one or more blank frames; and
identifying at least one active marker of the one or more active markers by one or more processors determining that the data correspond with the pattern.

2. The method of claim 1 , wherein the synchronizing further includes use of a phase locked loop to calibrate the signals of the signal controller and the rate of capture of the one or more sensor devices.

3. The method of claim 1 , further comprising defining the pattern as a unique signature for the one or more active markers.

4. The method of claim 1 , further comprising:
filtering out interfering electromagnetic radiation captured in frames that differ from the pattern.

5. The method of claim 1 , wherein determining that the data corresponds to the pattern includes predicting, for respective frames, that the electromagnetic radiation is to be represented as being present or absent based on the pattern, and confirming that the electromagnetic radiation is represented as being present or absent in the respective frames.

6. The method of claim 1 , further comprising,
synchronizing the rate of capture of the one or more sensor devices with a shutter action of a picture camera recording the live action scene, wherein the rate of capture of the one or more sensor devices is faster than the shutter action of the picture camera, such that the pattern is detectable within a single cycle of the picture camera.

7. The method of claim 1 , further comprising:
associating labels with the identified at least one active marker by at least one of an object and part of the object to which the at least one active marker is attached.

8. The method of claim 7 , further comprising:
transmitting the labels to a computer graphics rendering system to be mapped to a virtual model of the object.

9. A data capture system to recognize active markers in a live action scene for performance capture associated with a virtual production, the system comprising:
a signal controller to transmit signals to instruct one or more active markers to emit electromagnetic radiation according to a pattern including a sequence of electromagnetic radiation pulses and gap periods between the pulses, during specified time slots in which a presence of the electromagnetic radiation coincides with one or more predefined capture frames and an absence of the electromagnetic radiation coincides with one or more predefined blank frames of one or more sensor devices;
the one or more active markers to emit the electromagnetic radiation according to the pattern;
the one or more sensor devices to generate one or more capture frames representing captured electromagnetic radiation and one or more blank frames representing absence of electromagnetic radiation, wherein a rate of capture of the one or more sensor devices is synchronized with the signal controller; and
a computing device to identify at least one active marker of the one or more active markers based, at least in part, on determining that data associated with the one or more capture frames and the one or more blank frames, correspond with the pattern.

10. The system of claim 9 , wherein the signal controller and the one or more sensor devices are synchronized by determining a reference time according to a time code that corresponds with a reference frame of the one or more sensor devices and the one or more active markers are instructed by calibration signals of the signal controller to emit electromagnetic radiation based on the reference time.

11. The system of claim 9 , further comprising:
a picture camera device having a shutter action that is synchronized with a frame rate of the one or more sensor devices, wherein the frame rate of the one or more sensor devices is faster than the shutter action of the picture camera device, such that the pattern is detectable within a single frame of the picture camera device.

12. The system of claim 9 , further comprising a control unit in electrical communication with the one or more active markers, to receive the signals and control the one or more active markers to emit electromagnetic radiation according to the pattern.

13. The system of claim 12 , further comprising a wired strand coupled to the control unit and each active marker of the one or more active markers.

14. The system of claim 9 , wherein the computing device if further to associate labels with the identified at least one active marker by at least one of an object and part of the object to which the at least one active marker is attached.

15. A non-transitory computer-readable storage medium carrying program instructions thereon to recognize active markers in a live action scene for performance capture associated with a virtual production, the instructions when executed by one or more processors cause the one or more processors to perform operations comprising:
defining a pattern for one or more active markers to emit electromagnetic radiation as a unique signature, including a sequence of electromagnetic radiation pulses and gap periods between the pulses, during specified time slots in which a presence of the electromagnetic radiation coincides with one or more predefined capture frames and an absence of the electromagnetic radiation coincides with one or more predefined blank frames of one or more sensor devices;
transmitting the pattern to a signal controller to instruct the one or more active markers to emit the electromagnetic radiation according to the pattern;

generating data associated with the presence of electromagnetic radiation represented in the one or more capture frames and absence of electromagnetic radiation represented in the one or more blank frames; and
identifying at least one active marker of the one or more active markers based, at least in part, on determining that the data correspond with the pattern.

16. The computer-readable storage medium of claim 15 , wherein the operations further comprise:
determining a reference time according to a time code and transmitting the reference time to the signal controller and performance capture system to synchronize the electromagnetic radiation with the reference time.

17. The computer-readable storage medium of claim 15 , further comprising:
filtering out interfering electromagnetic radiation captured in frames that differ from the pattern.

18. The computer-readable storage medium of claim 15 , wherein the operations further comprise:
associating labels with the identified at least one active marker by at least one of an object and part of the object to which the at least one active marker is attached.

19. The computer-readable storage medium of claim

Assignees

• Weta Digital Ltd

Inventors

• Momcilovic Dejan
• Botting Jake

Classifications

• G06V10/141

  Control of illumination · CPC title

• G06T7/246 Primary

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06V10/143

  Sensing or illuminating at different wavelengths · CPC title

• G06V40/23

  Recognition of whole body movements, e.g. for sport training · CPC title

• H04N23/73

  by influencing the exposure time · CPC title