Multi-presence detection for performance capture

Patent metadata

Field	Value
Publication number	US-11308644-B2
Application number	US-202117378595-A
Country	US
Kind code	B2
Filing date	Jul 16, 2021
Priority date	Aug 28, 2020
Publication date	Apr 19, 2022
Grant date	Apr 19, 2022

Abstract

The present description relates relate to recalibration of a sensor device for performance capture, by detecting a miscalibration problem with sensor device and assessment of the problem. A recalibration system includes sensor devices initially calibrated at a recording site. A recording site change occurs and afterwards, a failure to match virtual rays projected from one sensor device with virtual rays projected from the active marker is detected. In response to determining the failure, the active marker is signaled to emit a unique display of light. The failure of the rays to match is assessed based on whether sensor devices capture the unique display of light. Three-dimensional (3-D) coordinates of an active marker is reconstructed from marker data of the calibrated sensor devices. A problematic sensor device is recalibrated based on the assessment, using the 3-D coordinates of the active marker from marker data of the remaining calibrated sensor devices, without stopping the recording.

First claim

We claim:

1. A method for recalibrating a sensor device to capture light from an active marker during a recording for a virtual production, the method comprising:
providing a first sensor device, a second sensor device, and a third sensor device, each of the sensor devices being initially calibrated at a recording site to capture marker data for three-dimensional (3-D) coordinates of an active marker;
after a recording site change, determining a failure of virtual rays projected from the first sensor device to match with virtual rays projected from the active marker;
in response to determining the failure, signaling for the active marker to emit a unique display of light;
assessing the failure based on at least one of the first sensor device, the second sensor device, or the third sensor device capturing the unique display of light; and
without stopping the recording, recalibrating the first sensor device based on the assessment, using 3-D coordinates of the active marker from marker data of the second sensor device and the third sensor device.

2. The method of claim 1 , wherein assessing the failure includes:
detecting that the first sensor device, the second sensor device, and the third sensor device each capture the unique display of light; and
determining that the first sensor device is miscalibrated and the active marker is in a field of view of the first sensor device.

3. The method of claim 2 , wherein the recording site change includes the first sensor device being moved to a changed physical position.

4. The method of claim 3 , wherein the recalibrating includes calculating the changed physical position of the first sensor device based by applying the 3-D coordinates of the active marker and corresponding two-dimensional (2-D) coordinates of the first sensor device to a perspective-n-point pose problem formula.

5. The method of claim 1 , wherein assessing the failure includes:
detecting that the second sensor device or the third sensor device captures the unique display of light and the first sensor device fails to capture the unique display of light; and
repositioning the first sensor device to a changed physical position to have a field of view region including the active marker and overlapping with a field of view of the second sensor device or the third sensor device, thereby causing the first sensor device to be miscalibrated.

6. The method of claim 5 , wherein the recording site change includes an initial field of view of the first sensor device in an initial sensor position becoming at least partially occluded.

7. The method of claim 1 , wherein the recording site change is a modification of an intrinsic parameter of the first sensor device.

8. The method of claim 1 , wherein assessing the failure further includes:
identifying the active marker by the marker data representing the unique display of light emitted by the active marker and captured in particular frames by the first sensor device, the second sensor device, and the third sensor device, when each sensor device is initially calibrated.

9. A system for recalibrating a sensor device to capture active marker light during a recording associated with a virtual production, the system comprising:
a first sensor device, a second sensor device, and a third sensor device, each of the sensor devices being initially calibrated at a recording site to capture marker data for three-dimensional (3-D) coordinates of an active marker;
a plurality of active markers;
at least one computing device comprising one or more processors and a tangible processor-readable storage device including instructions for:
after a recording site change, determining a failure of virtual rays projected from the first sensor device to match with virtual rays projected from the active marker;
in response to determining the failure, signaling for the active marker to emit a unique display of light;
assessing the failure based on at least one of the first sensor device the second sensor device, or the third sensor device capturing the unique display of light; and
without stopping the recording, recalibrating the first sensor device based on the assessment, using 3-D coordinates of the active marker from marker data of the second sensor device and the third sensor device.

10. The system of claim 9 , wherein assessing the failure includes:
detecting the first sensor device, the second sensor device, and the third sensor device each capture the unique display of light; and
determining that the first sensor device is miscalibrated and the active marker is in a field of view of the first sensor device.

11. The system of claim 9 , wherein the recording site change includes the first sensor device being moved to a changed physical position, and wherein the recalibrating includes calculating the changed physical position of the first sensor device based by applying the 3-D coordinates of the active marker and corresponding two-dimensional (2-D) coordinates of the first sensor device to a perspective-n-point pose problem formula.

12. The system of claim 9 , wherein the recording site change includes a field of view of the first sensor device in an initial sensor position becoming at least partially occluded and wherein assessing the failure further comprises:
detecting that the second sensor device or the third sensor device captures the unique display of light and the first sensor device fails to capture the unique display of light; and
repositioning the first sensor device to a changed physical position to have a field of view region including the active marker and overlapping with a field of view of the second sensor device or the third sensor device, thereby causing the first sensor device to be miscalibrated.

13. The system of claim 9 , wherein the recording site change is a modification of an intrinsic parameter of the first sensor device.

14. The system of claim 9 , wherein assessing the failure further includes:
identifying the active marker by the marker data representing the unique display of light emitted by the active marker and captured in particular frames by the first sensor device and second sensor device.

15. A non-transitory computer-readable storage medium carrying program instructions thereon for recalibrating a sensor device to capture active marker light during a recording associated with a virtual production, the program instructions when executed by one or more processors cause the one or more processors to perform operations comprising:
receiving initial calibration information for a first sensor device, a second sensor device, and a third sensor device, each of the sensor devices being initially calibrated at a recording site to capture marker data for three-dimensional (3-D) coordinates of an active marker;
after a recording site change, determining a failure of virtual rays projected from the first sensor device to match with virtual rays projected from the active marker;
in response to determining the failure, signaling for the active marker to emit a unique display of light;
assessing the failure based on at least one of the first sensor device, the second sensor device, or the third sensor device capturing the unique display of light; and
without stopping the recording, recalibrating the first sensor device based on the assessment, using 3-D coordinates of the active marker from marker data of the second sensor device and the third sensor device.

16. The computer-readable storage medium of claim 15 , wherein assessing the failure includes:
detecting the first sensor device, the second sensor device, and the th

Assignees

• Weta Digital Ltd

Inventors

• Momcilovic Dejan
• Botting Jake

Classifications

• G06T2207/30196

  Human being; Person · CPC title

• G06T7/73

  using feature-based methods · CPC title

• G06T7/246

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06T2207/30204

  Marker · CPC title

• G06T7/80 Primary

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title