Electronic device and method for spatial synchronization of videos

Patent metadata

Field	Value
Publication number	US-12167150-B2
Application number	US-202117515657-A
Country	US
Kind code	B2
Filing date	Nov 1, 2021
Priority date	Nov 1, 2021
Publication date	Dec 10, 2024
Grant date	Dec 10, 2024

Abstract

An electronic device is provided that determines initial three-dimensional (3D) coordinates of a lighting device. The electronic device controls an emission of light from the lighting device based on control signals. The emitted light includes at least one of a pattern of alternating light pulses or a continuous light pulse. The electronic device controls a plurality of imaging devices to capture a first plurality of images that include information about the emitted light. Based on the determined initial 3D coordinates and the information about the emitted light included in the first plurality of images, the electronic device estimates a plurality of rotation values and a plurality of translation values of each imaging device. Based on the plurality of rotation values and the plurality of translation values, the electronic device applies a simultaneous localization and mapping process for each imaging device, for spatial synchronization of the plurality of imaging devices.

First claim

What is claimed is:

1. An electronic device, comprising:
circuitry communicatively coupled to a plurality of imaging devices and a lighting device, wherein the circuitry is configured to:
determine initial three-dimensional (3D) coordinates of the lighting device, wherein the lighting device includes a grid of lights and an edge light;
control an emission of light from the lighting device based on at least one control signal, wherein the emitted light includes at least one of a pattern of alternating light pulses or a continuous light pulse;
control the plurality of imaging devices to capture a first plurality of images that include information about the emitted light;
estimate a plurality of rotation values and a plurality of translation values of each imaging device, based on the determined initial 3D coordinates of the lighting device and the information about the emitted light;
determine a first set of images of the first plurality of images that includes information about the pattern of alternating light pulses of the emitted light;
determine a center of each light of the grid of lights of the lighting device in a first set of frames of the first set of images, wherein the first set of frames includes an ON pattern of the pattern of alternating light pulses;
estimate, based on the initial 3D coordinates of the lighting device and the determined center of each light of the grid of lights in the first set of frames, a first rotation value and a first translation value of the plurality of rotation values and the plurality of translation values for each imaging device of the plurality of imaging devices; and
apply a simultaneous localization and mapping (SLAM) process for each imaging device of the plurality of imaging devices, based on the plurality of rotation values and the plurality of translation values, for spatial synchronization of the plurality of imaging devices.

2. The electronic device according to claim 1 , wherein the circuitry is further configured to:
generate a synchronization signal that includes a preamble pulse and a sequence of alternating ON and OFF pulse, wherein the at least one control signal includes the synchronization signal; and
control the emission of light from the lighting device based on the generated synchronization signal that includes the pattern of alternating light pulses corresponding to the generated synchronization signal.

3. The electronic device according to claim 2 , wherein
the circuitry is further configured to generate the synchronization signal based on a set of parameters corresponding to each imaging device of the plurality of imaging devices, and
the set of parameters includes at least a frame rate of each imaging device of the plurality of imaging devices.

4. The electronic device according to claim 2 , wherein for the control of the emission of the light from the lighting device based on the generated synchronization signal, the circuitry is further configured to:
activate the grid of lights of the lighting device to generate an ON pattern of the pattern of alternating light pulses of the emitted light;
deactivate the grid of lights of the lighting device to generate an OFF pattern of the pattern of alternating light pulses of the emitted light; and
deactivate the edge light of the lighting device.

5. The electronic device according to claim 1 , wherein the circuitry is further configured to control the plurality of imaging devices to capture the first set of images based on a determination that the lighting device is in a field-of-view of a respective imaging device of the plurality of imaging devices.

6. The electronic device according to claim 1 , wherein the circuitry is further configured to:
apply a set of post-processing operations on the first set of frames of the first plurality of images; and
determine the center of each light of the grid of lights of the lighting device based on the applied set of post-processing operations on the first set of frames.

7. The electronic device according to claim 1 , wherein the circuitry is further configured to:
apply a neural network model on the first set of frames of the first plurality of images to determine a first frame, wherein the first frame includes the information about the pattern of alternating light pulses; and
determine the center of each light of the grid of lights of the lighting device in the determined first frame of the first set of frames.

8. The electronic device according to claim 1 , wherein the circuitry is further configured to:
control the lighting device to activate the edge light of the lighting device;
control a transformation of the lighting device towards each imaging device of the plurality of imaging devices, wherein the transformation includes at least one of a rotation or a translation of the lighting device;
control, based on the transformation of the lighting device, the plurality of imaging devices to capture light emitted by the edge light;
receive a second plurality of images from the plurality of imaging devices, wherein the received second plurality of images includes information about the light emitted by the edge light; and
estimate a slope of the information about the light emitted by the edge light in the second plurality of images.

9. The electronic device according to claim 8 , wherein the circuitry is further configured to:
determine, based on the estimated slope of the information about the light emitted by the edge light in the second plurality of images, a set of grid lines that pass through the determined center of each light of the grid of lights of the lighting device in the first set of frames;
determine, based on an intersection of the determined set of grid lines, at least one projected 2D position of the center of each light in the first set of frames; and
estimate, based on the initial 3D coordinates of the lighting device and the determined at least one projected 2D position of the center of each light, the first rotation value and the first translation value of the plurality of rotation values and the plurality of translation values of each imaging device of the plurality of imaging devices.

10. The electronic device according to claim 1 , wherein the circuitry is further configured to estimate, based on a perspective-n-point (PnP) technique, the first rotation value and the first translation value for each imaging device of the plurality of imaging devices.

11. The electronic device according to claim 9 , wherein the circuitry is further configured to determine, based on a mathematical optimization function, the set of grid lines that pass through the determined center of each light of the grid of lights in the first set of frames.

12. The electronic device according to claim 9 , wherein the circuitry is further configured to determine, based on a neural network model, the set of grid lines that pass through the determined center of each light of the grid of lights in the first set of frames.

13. The electronic device according to claim 1 , wherein the circuitry is further configured to control the emission of the light from the lighting device based on a specific control signal, wherein
the emitted light includes the continuous light pulse corresponding to the specific control signal, and
the at least one control signal includes the specific control signal.

14. The electronic device according to claim 13 , wherein, the circuitry is further configured to:
for the control of the emission of the light from the lighting device based on the specific control signal:
activate the grid of lights of the lighting device to emit the light; and

Assignees

• Sony Group Corp
• Sony Corp America

Inventors

• Faust Brent
• Liu Cheng-Yi

Classifications

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10152

  Varying illumination · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• H04N17/002

  for television cameras · CPC title