Resolving three dimensional spatial information using time-shared structured lighting that embeds digital communication

Patent metadata

Field	Value
Publication number	US-9930320-B2
Application number	US-201615146454-A
Country	US
Kind code	B2
Filing date	May 4, 2016
Priority date	May 4, 2016
Publication date	Mar 27, 2018
Grant date	Mar 27, 2018

Abstract

Systems, methods, and computer readable media to resolve three dimensional spatial information of cameras used to construct 3D images. Various embodiments perform communication synchronization between a first image capture system and one or more other image capture systems and generate a first flash pulse that projects a light pattern into an environment. An image is captured that includes the light pattern and a modulated optical signal encoded with an identifier of one of the first image capture system and related-camera information. A second flash from another image capture systems may flash at a second time based on the communication synchronization. During the second flash, the first image capture system captures a second image of the environment. Based on the first and second images, the first image capture system determines the orientation of the second image capture system relative to the first image capture system.

First claim

The invention claimed is:

1. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause one or more programmable control devices to:
obtain, by a first image capturing system, communication synchronization with a second image capturing system;
generate, by the first image capturing system, a first flash pulse that projects a first light pattern into a scene at a first time, wherein the first flash pulse includes a modulated optical signal that encodes an image capturing system identifier of the first image capturing system;
capture, by the first image capturing system, a first image of the scene at the first time;
determine, by the first image capturing system, the second image capturing system will generate a second flash pulse at a second time based on the communication synchronization, wherein the second flash pulse projects a second light pattern into the scene;
capture, by the first image capturing system, a second image of the scene at the second time; and
determine the orientation of the second image capturing system, relative to the first image capturing system, based on the first and the second image.

2. The non-transitory program storage device of claim 1 , wherein the modulated optical signal includes a modulated amplitude of the first flash pulse to encode the image capturing system identifier of the first image capturing system.

3. The non-transitory program storage device of claim 1 , wherein the instructions to cause the one or more programmable control devices to capture the second image of the scene at the second time comprise instructions to cause the one or more programmable control devices to decode a second optical signal emitted by the second image capturing system.

4. The non-transitory program storage device of claim 3 , wherein the instructions to cause the one or more programmable control devices to decode the second optical signal comprise instructions to cause the one or more programmable control devices to decode the second optical signal to determine a focal length of the second image capturing system.

5. The non-transitory program storage device of claim 4 , wherein the instructions further cause the one or more programmable control devices to determine, by the first image capture system, a location of the second image capture system in the scene based on the first image, the second image, and the focal length.

6. The non-transitory program storage device of claim 1 , wherein the instructions further cause the one or more programmable control devices to determine a location of the at least one image capturing system in the scene based on the first and the second image and maintain position tracking relative to the at least one image capturing system using one or more motion sensors.

7. The non-transitory program storage device of claim 1 , wherein the modulated optical signal modulates a time delay in reference to a timestamp for the optical signal to encode the image capturing system identifier of the first image capturing system.

8. The non-transitory program storage device of claim 1 , wherein the instructions to cause the one or more programmable control devices to determine the relative orientation of the second image capturing system comprise instructions to cause the one or more programmable control devices to compare the differences between the first image and the second image based on a least error type algorithm.

9. The non-transitory program storage device of claim 1 , wherein the instructions to cause the one or more programmable control devices to obtain communication synchronization with the second image capturing system comprise instructions to cause the one or more programmable control devices to generate a series of flash events to implement a network multiple-access protocol that obtains communication synchronization with the second image capturing system.

10. The non-transitory program storage device of claim 1 , wherein the instructions to cause the one or more programmable control devices to capture the first image of the scene at the first time comprise instructions to cause the one or more programmable control devices to decode the modulated optical signal by decoupling a carrier pulse frequency information from the image capturing system identifier of the first image capturing system.

11. A method for coordinating devices in a structured light system, comprising:
obtaining, by a first image capture system, communication synchronization with a second image capture system;
generating, by the first image capture system, a first flash pulse that projects a light pattern into an environment at a first time, wherein the first flash pulse includes a modulated first flash pulse that encodes a system identifier of the first image capture system;
capturing, by the first image capture system, a first image of the environment at the first time;
determining, by the first image capture system, the second image capture system will flash at a second time based on the communication synchronization;
capturing, by the first image capture system, a second image of the environment at the second time; and
determining, by the first image capture system, an orientation of the second image capture system relative to the first image capture system based on the first and second images.

12. The method of claim 11 , wherein the modulated first flash pulse includes a modulated amplitude of the first flash pulse to encode the system identifier of the first image capture system.

13. The method of claim 11 , wherein capturing a second image of the environment at the second time further comprises decoding, by the first image capture system, a second flash pulse generated at the second time to identify the second image capture system and a focal length of the second image capture system.

14. The method of claim 13 , wherein determining the orientation of the second image capture system further comprises determining, by the first image capture system, a location of the second image capture system in the environment based on the first image, the second image, and the focal length and maintaining position tracking relative to the second image capture system using one or more motion sensors.

15. The method of claim 11 , wherein determining the orientation of the second image capture system further comprises comparing the differences between the first image and the second image using a least error type algorithm.

16. A system comprising:
a structured-light source;
an image capture device coupled to the structured-light source;
a memory; and
one or more programmable control devices operable to interact with the structured-light source, the image capture device, and the memory to perform operations comprising:
synchronizing, using the image capture device, with a second image capture device;
generating, using the image capture device, a first flash pulse from the structured-light source that projects a light pattern into a scene at a first image capture event, wherein the first flash pulse includes a modulated optical signal that encodes a device identifier of the image capture device;
capturing, by the image capture device, a first image of the scene at the first image capture event;
determining by the image capture device that the second image capture device will flash at a second image capture event based on the synchronization;
capturing, using the image capture device, a second image of the scene at the second capturing event, wherein the second image includes a second light pattern projected i

Assignees

• Apple Inc

Inventors

• Chen Denis G
• Lin Chin Han

Classifications

• H04N13/0253

  Electricity · mapped topic

• H04N13/0203

  Electricity · mapped topic

• H04N13/0271

  Electricity · mapped topic

• H04N13/0296 Primary

  Electricity · mapped topic

• H04N13/239 Primary

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title