Video monitoring using plenoptic cameras and mirrors

Patent metadata

Field	Value
Publication number	US-10044985-B1
Application number	US-201213656148-A
Country	US
Kind code	B1
Filing date	Oct 19, 2012
Priority date	Oct 19, 2012
Publication date	Aug 7, 2018
Grant date	Aug 7, 2018

Abstract

A video-based monitoring and image post-processing system may include a plenoptic camera that has multiple mirrors in its field of view. The plenoptic camera may capture four-dimensional light-field information for multiple images of its field of view and may embed that information in image data sent to a local or backend server for post-processing. An image processing application on the server may extract image data for pixels that collectively represent one of the mirrors, and apply a focusing operation on the extracted image data to generate a video stream of a virtual camera represented by the mirror, dependent on the image information and the distance between the mirror and the camera. The application may correct distortions due to the curvature of the mirror. The system may provide video monitoring in areas without a single line of sight without the need for a large number of expensive cameras.

First claim

What is claimed is:

1. A system, comprising:
a plenoptic camera located in a materials handling facility, wherein the plenoptic camera has within its original field of view two or more mirrors, and wherein at least a portion of a respective area of the materials handling facility that is in a field of view of each of the two or more mirrors is not in the original field of view of the plenoptic camera; and
a server configured to communicate with the plenoptic camera;
wherein the plenoptic camera is configured to:
capture raw frames of a first video stream, wherein each raw frame includes image data for a scene including the two or more mirrors in the original field of view of the plenoptic camera; and
send the raw frames to the server for post-processing;

wherein the server is further configured to:
for each raw frame, post-process the image data in the raw frame to generate respective frames for each of two or more new video streams corresponding to two or more virtual cameras, wherein the two or more new video streams are generated from the first video stream, and wherein each virtual camera depicts an image reflected in a respective one of the two or more mirrors from a perspective of the plenoptic camera;
wherein, to post-process the image data in the raw frame to generate a frame for a video stream of a virtual camera, the server is configured to apply a focusing operation to a portion of the image data in the raw frame that represents image pixels that collectively depict a respective one of the two or more mirrors, wherein the generated frame depicts a portion of the raw frame that comprises the respective one of the two or more mirrors and does not comprise another one of the two or more mirrors, and wherein the focusing operation is dependent on a distance between the respective mirror and the plenoptic camera; and
store the two or more new video streams corresponding to the two or more virtual cameras, wherein the two or more video new streams comprise data representing the generated respective frames each having different portions of the original field of view.

2. The system of claim 1 , wherein, to post-process the image data in the raw frame to generate a frame for a video stream of a virtual camera, the server is further configured to extract the portion of the image data that represents image pixels that collectively depict the given mirror prior to applying the focusing operation.

3. The system of claim 1 , wherein, to post-process the image data in the raw frame to generate a frame for a video stream of a virtual camera, the server is further configured to:
apply the focusing operation to the image data for the entire raw frame to produce focused image data; and
extract the portion of the focused image data that represents image pixels that collectively depict the given mirror.

4. A method, comprising,
capturing, over a period of time, a first video stream comprising image information for multiple images of an area that is in an original field of view of a plenoptic camera, wherein the area includes two or more mirrors on each of which is reflected an image of a respective portion of another area that is in a field of view of the respective mirror, and wherein at least a portion of the other area is not in the original field of view of the plenoptic camera;
for each one of the multiple images, processing the image information to generate two or more images representing views of respective portions of the other area from perspectives of two or more virtual cameras that each have the same field of view as a respective one of the two or more mirrors, wherein each generated image depicts a portion of the one of the multiple images that comprises a respective one of the two or more mirrors but does not depict all of the area that is in the original field of view of the plenoptic camera; and
storing two or more new video streams for the two or more virtual cameras represented by the two or more mirrors, wherein the two or more new video streams are generated from the first video stream, and wherein the new video streams comprise data representing the respective generated images each having different portions of the original field of view.

5. The method of claim 4 , wherein said processing the image information to generate two or more images comprises applying a focusing operation to respective portions of the image information that represent image pixels that collectively depict respective ones of the two or more mirrors, and wherein the focusing operation is dependent on the distances between the mirrors and the plenoptic camera.

6. The method of claim 4 , wherein at least one of the mirrors is curved or spherical, and wherein said processing comprises applying a correction operation to the image information to compensate for a distortion of the reflected image caused by the shape of the curved or spherical mirror or to reverse the reflected image.

7. The method of claim 4 , wherein said capturing image information for multiple images of the first video stream comprises capturing the first video stream using a video camera.

8. The method of claim 4 , wherein said capturing image information for multiple images of the first video stream comprises capturing a sequence of images using a still camera.

9. The method of claim 4 , wherein said processing the image information to generate two or more images comprises extracting respective portions of the image information that represent image pixels that collectively depict respective ones of the two or more mirrors.

10. The method of claim 9 , wherein said extracting is dependent on a pre-determined mapping between the image pixels that collectively depict the mirrors and identifiers of the mirrors.

11. The method of claim 9 , wherein said processing further comprises, prior to said extracting:
analyzing the image information; and
detecting, dependent on said analyzing, respective portions of the image information that represent the image pixels that collectively depict respective ones of the two or more mirrors.

12. A non-transitory, computer-readable storage medium, storing program instructions that when executed on one or more computers cause the one or more computers to perform:
accessing, from a first video stream, image data that includes four-dimensional light-field information about a scene, wherein the four-dimensional light-field information comprises information in two planar dimensions and information in two angular dimensions, wherein the scene includes objects that are at different distances from a camera from which the image data was obtained, wherein the scene includes two or more mirrors in each of which an image of a respective portion of an area is reflected, and wherein at least a portion of the reflected area is not visible in the scene other than in the mirrors;
processing the image data to generate two or more images that represent views of respective portions of the area from perspectives of two or more virtual cameras that each have the same field of view as a respective one of the two or more mirrors, wherein each generated image depicts a portion of the scene but does not depict the entire scene, wherein said processing comprises applying a focusing operation to the image data such that each generated image is in focus, and wherein the focusing operation is dependent on distances between the mirrors and the camera from which the image data was obtained; and
storing two or more new video streams for the two or more virtual cameras represented by the two or more mirrors, wherein the two or more new video streams are generated from the first video stream, and wherein the new video streams c

Assignees

• Amazon Tech Inc

Inventors

• Parker Erik R

Classifications

• H04N7/181 Primary

  for receiving images from a plurality of remote sources · CPC title

• H04N25/41

  Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors · CPC title

• H04N23/58

  Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors · CPC title

• H04N23/957

  Light-field or plenoptic cameras or camera modules · CPC title

• H04N9/12

  Picture reproducers (H04N9/11 takes precedence) · CPC title