Panoramic camera systems

Patent metadata

Field	Value
Publication number	US-10368062-B2
Application number	US-201715490889-A
Country	US
Kind code	B2
Filing date	Apr 18, 2017
Priority date	Apr 13, 2017
Publication date	Jul 30, 2019
Grant date	Jul 30, 2019

Abstract

A camera system captures images from a set of cameras to generate binocular panoramic views of an environment. The cameras are oriented in the camera system to maximize the minimum number of cameras viewing a set of randomized test points. To calibrate the system, matching features between images are identified and used to estimate three-dimensional points external to the camera system. Calibration parameters are modified to improve the three-dimensional point estimates. When images are captured, a pipeline generates a depth map for each camera using reprojected views from adjacent cameras and an image pyramid that includes individual pixel depth refinement and filtering between levels of the pyramid. The images may be used generate views of the environment from different perspectives (relative to the image capture location) by generating depth surfaces corresponding to the depth maps and blending the depth surfaces.

First claim

What is claimed is:

1. A method comprising:
receiving, at an image processing system comprising a long term storage and a memory, a request to generate depth maps for each frame of a sequence of frames, wherein each frame comprises image content from a plurality of cameras of an image capture system stored in the long term storage of the image processing system;
wherein a pipeline comprising a series of steps is used to generate a depth map from image content, each step of the pipeline generating a result component based on set of input components, wherein the pipeline comprises:
an image content processing step generating processed image content for a frame based on input image content associated with the frame; and
a depth generation step generating a depth map for a frame based on input processed image content associated with the frame;
and wherein at least one step of the pipeline uses input components from a plurality of frames;

generating a first depth map for a current frame using the pipeline, wherein components used to generate the first depth map are marked in the memory;
determining a set of unmarked components stored in the memory and removing the set of unmarked components from memory;
determining a set of marked components stored in the memory and unmarking each component of the set of unmarked components; and
advancing the current frame to the next frame in the sequence of frames.

2. The method of claim 1 , wherein generating a first depth map for a current frame using the pipeline comprises recursively calling to generate the depth map for a current frame of the sequence of frames, each recursive call generating a subject component of the depth map for the current frame by performing the steps of:
determining a set of precursor components needed to generate the subject component;
checking the memory of the image processing system for each precursor component;
responsive to a precursor component being present in the memory, marking the precursor component in memory;
responsive to a precursor component not being present in the memory:
recursively calling to generate the missing precursor component; and
marking the generated precursor component in memory
generating, at the image processing system, the subject component using the set of precursor components; and
returning to any parent recursive call.

3. The method of claim 1 , wherein the image content processing step comprises filtering the input image content.

4. The method of claim 3 , wherein filtering the input image content for the image content processing step comprises using a time dependent self guided bilateral filter to filter the image content based on image content from adjacent frames.

5. The method of claim 1 , wherein the image content comprises raw or minimally processed data from the image sensors of the cameras of the image capture system.

6. The method of claim 5 , wherein the pipeline further comprises:
a raw image content processing step generating processed raw image content for a frame based on input raw image content associated with the frame; and
a raw image content conversion step generating image content in a standard format for a frame based on input processed raw image content associated with the frame.

7. The method of claim 1 , further comprising:
dividing the sequence of frames into a plurality of subsequences;
based on the total time dependency of the pipeline, adding additional frames to each of the subsequences to ensure depth map consistency between the sequences.

8. A system comprising:
an image processing system comprising a long term storage and a memory, the image processing system configured to:
receive a request to generate depth maps for each frame of a sequence of frames, wherein each frame comprises image content from a plurality of cameras of an image capture system stored in the long term storage of the image processing system;
generate a first depth map for a current frame using the pipeline, wherein components used to generate the first depth map are marked in the memory;
determine a set of unmarked components stored in the memory and removing the set of unmarked components from memory;
determine a set of marked components stored in the memory and unmarking each component of the set of unmarked components; and
advance the current frame to the next frame in the sequence of frames;

wherein a pipeline comprising a series of steps is used to generate a depth map from image content, each step of the pipeline generating a result component based on set of input components, wherein the pipeline comprises:
an image content processing step generating processed image content for a frame based on input image content associated with the frame; and
a depth generation step generating a depth map for a frame based on input processed image content associated with the frame; and

wherein at least one step of the pipeline uses input components from a plurality of frames.

9. The system of claim 8 , wherein generating a first depth map for a current frame using the pipeline comprises recursively calling to generate the depth map for a current frame of the sequence of frames, each recursive call generating a subject component of the depth map for the current frame by performing the steps of:
determining a set of precursor components needed to generate the subject component;
checking the memory of the image processing system for each precursor component;
responsive to a precursor component being present in the memory, marking the precursor component in memory;
responsive to a precursor component not being present in the memory:
recursively calling to generate the missing precursor component; and
marking the generated precursor component in memory
generating, at the image processing system, the subject component using the set of precursor components; and
returning to any parent recursive call.

10. The system of claim 8 , wherein the image content processing step comprises filtering the input image content.

11. The system of claim 10 , wherein filtering the input image content for the image content processing step comprises using a time dependent bilateral filter to filter the image content based on image content from adjacent frames.

12. The system of claim 8 , wherein the image content comprises raw or minimally processed data from the image sensors of the cameras of the image capture system.

13. The system of claim 12 , wherein the pipeline further comprises:
a raw image content processing step generating processed raw image content for a frame based on input raw image content associated with the frame; and
a raw image content conversion step generating image content in a standard format for a frame based on input processed raw image content associated with the frame.

14. The system of claim 9 , wherein the image processing system is further configured to:
divide the sequence of frames into a plurality of subsequences;
based on the total time dependency of the pipeline, add additional frames to each of the subsequences to ensure depth map consistency between the sequences.

15. A non-transitory computer readable storage medium comprising instructions which, when executed by a processor, cause the processor to perform the steps of:
receiving, at an image processing system comprising a long term storage and a memory, a request to generate depth maps for each frame of a sequence of frames, wherein each frame comprises image content from a plurality of cameras of an image capture system stored in the long term storage of the image processing system;
wherein a pipeli

Assignees

• Facebook Inc

Inventors

• Briggs Forrest Samuel
• Toksvig Michael John
• Cabral Brian Keith

Classifications

• G06V30/1914

  Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries, e.g. user dictionaries · CPC title

• G06V30/19147

  Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/75

  Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/2411

  based on the proximity to a decision surface, e.g. support vector machines · CPC title