Multi-perspective imaging system and method

Patent metadata

Field	Value
Publication number	US-10482594-B2
Application number	US-201715712600-A
Country	US
Kind code	B2
Filing date	Sep 22, 2017
Priority date	Sep 23, 2016
Publication date	Nov 19, 2019
Grant date	Nov 19, 2019

Abstract

A system includes a processor to capture a first image of a scene by a first imaging device of the array of imaging devices, capture a second image of the scene by a second imaging device of the array of imaging devices, perform image analysis of the first image and the second image and determining that an object is present in the first image and the object is present in the second image, the first image representing a first perspective of the object and the second image representing a second perspective of the object different from the first perspective of the object, classify the object with a classification based on a list of known objects and weight an object portion of the first image and an object portion of the second image based on the classification, compress the first image and the second image based on the weighting, encode the first image and the second image as multi-perspective image data, and store the multi-perspective image data in the non-transitory computer-readable medium based on the classification.

First claim

What is claimed is:

1. A system, comprising:
an array of imaging devices;
a processor; and
a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the processor to perform operations comprising:
capturing a first image of a scene by a first imaging device of the array of imaging devices;
capturing a second image of the scene by a second imaging device of the array of imaging devices;
performing image analysis of the first image and the second image and determining that an object is present in the first image and the object is present in the second image, the first image representing a first perspective of the object and the second image representing a second perspective of the object different from the first perspective of the object;
classifying the object with a classification based on a list of known objects and weighting an object portion of the first image and an object portion of the second image based on the classification;
compressing the first image and the second image based on the weighting, the object portion of the first image and the object portion of the second image being compressed at a particular compression level based on at least one of the classification or the weighting;
encoding the first image and the second image as multi-perspective image data; and
storing the multi-perspective image data in the non-transitory computer-readable medium.

2. The system of claim 1 , the operations further comprising:
decoding the multi-perspective image data and displaying a representation of the scene using the first image and the second image.

3. The system of claim 1 , the operations further comprising:
compressing the first image and the second image at a compression level based on one or more of the classification and the weighting.

4. The system of claim 1 , the operations further comprising:
compressing the object portion of the first image and the object portion of the second image at a first particular compression level based on one or more of the classification and the weighting, and compressing a non-object portion of the first image and a non-object portion of the second image at a second particular compression level, the first particular compression level comprising a lossless compression level and the second particular compression level comprising a lossy compression level.

5. The system of claim 1 , the operations further comprising:
discarding the first image and the second image when the weighting does not meet a particular threshold.

6. The system of claim 1 , the system further comprising a sensor device, the operations further comprising:
determining a distance to the object from the first image device and the second imaging device using the sensor device to determine time-of-flight;
weighting the object portion of the first image and the object portion of the second image based on the distance; and
compressing the first image and the second image based on the weighting, and encoding the first image and the second image as the multi-perspective image data.

7. The system of claim 1 , the system further comprising a Global Positioning System (GPS) device, the operations further comprising:
determining a location of the object using the GPS device;
weighting the object portion of the first image and the object portion of the second image based on the location of the object; and
compressing the first image and the second image based on the weighting, and encoding the first image and the second image as the multi-perspective image data.

8. The system of claim 1 , the operations further comprising:
determining bidirectional reflectance distribution function (BRDF) information for the object based on the first image and the second image;
weighting the object portion of the first image and the object portion of the second image based on the BRDF information; and
compressing the first image and the second image based on the weighting, and encoding the first image and the second image as the multi-perspective image data.

9. The system of claim 1 , the operations further comprising:
activating the array of imaging devices to begin capturing images in response to user input, the user input comprising input to one or more of a touch screen device and a microphone.

10. The system of claim 1 , the operations further comprising:
activating the array of imaging devices to begin capturing images in response to user input, the user input comprising one or more of a gesture made on a touch screen and a verbal command.

11. The system of claim 1 , the operations further comprising:
activating the array of imaging devices to begin capturing images when the array of imaging devices determine that the object is present in one or more of the first image and the second image.

12. The system of claim 1 , the system further comprising a Global Positioning System (GPS) device, the operations further comprising:
determining a location using the GPS device;
activating the array of imaging devices to begin capturing images based on the location.

13. The system of claim 1 , the system further comprising a vehicle, the array of imaging devices mounted on an exterior of the vehicle.

14. The system of claim 1 , the system further comprising a vehicle, the array of imaging devices mounted on an interior of the vehicle.

15. The system of claim 1 , the operations further comprising:
generating a panoramic image of the object using the multi-perspective image data.

16. The system of claim 1 , the operations further comprising:
displaying a representation of the scene including displaying the object from a perspective including a front of a vehicle, a first side of the vehicle, a second side of the vehicle, and a rear of the vehicle.

17. The system of claim 1 , the operations further comprising:
displaying a representation of the scene including displaying the object from the first perspective of the object and transitioning to displaying the object from the second perspective of the object.

18. The system of claim 1 , the image analysis further comprising:
determining the object in the object portion of the first image is a same object as the object in the object portion of the second image by comparing the object portion of the first image and the object portion of the second image to features and patterns in images of objects in the list of known objects.

19. The system of claim 18 , the image analysis further comprising determining that the object in the object portion of the first image is the same object as the object in the object portion of the second image based on depth information, BRDF information, map data, and location information.

20. The system of claim 18 , the image analysis further comprising determining the object portion of the first image as a subset of the first image representing the object and determining the object portion of the second image as a subset of the second image representing the object.

21. A method comprising:
obtaining a first image of a scene captured by a first imaging device of an array of imaging devices;
obtaining a second image of the scene captured by a second imaging device of the array of imaging devices;
performing, by at least one processor, image analysis of the first image and the second image and determining that an object is present in the first image and the object is present in the second image, the first image representing a first perspective of the

Assignees

• Apple Inc

Inventors

• Potter Daniel E
• Zhang Arthur Y

Classifications

• G06T7/001 Primary

  using an image reference approach · CPC title

• H04N19/136

  Incoming video signal characteristics or properties · CPC title

• H04N19/102

  characterised by the element, parameter or selection affected or controlled by the adaptive coding · CPC title

• H04N19/17

  the unit being an image region, e.g. an object · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title