Technologies for adjusting a perspective of a captured image for display

Patent metadata

Field	Value
Publication number	US-9934573-B2
Application number	US-201414488516-A
Country	US
Kind code	B2
Filing date	Sep 17, 2014
Priority date	Sep 17, 2014
Publication date	Apr 3, 2018
Grant date	Apr 3, 2018

Abstract

Technologies for adjusting a perspective of a captured image for display on a mobile computing device include capturing a first image of a user by a first camera and a second image of a real-world environment by a second camera. The mobile computing device determines a position of an eye of the user relative to the mobile computing device based on the first captured image and a distance of an object in the real-world environment from the mobile computing device based on the second captured image. The mobile computing device generates a back projection of the real-world environment captured by the second camera to the display based on the determined distance of the object in the real-world environment relative to the mobile computing device, the determined position of the user's eye relative to the mobile computing device, and at least one device parameter of the mobile computing device.

First claim

The invention claimed is:

1. A mobile computing device for adjusting a perspective of a captured image for display, the mobile computing device comprising:
a permanently non-transparent display;
a camera system comprising a first camera and a second camera, the camera system to capture, while the user is holding the mobile computing device, (i) a first image of a user of the mobile computing device with the first camera and (i) a second image of a real-world environment of the mobile computing device with the second camera;
an eye tracking module to determine a position of an eye of the user relative to the mobile computing device based on the first captured image;
an object distance determination module to determine a distance of an object in the real-world environment relative to the mobile computing device based on the second captured image; and
an image projection module to (i) determine an angular size of the mobile computing device from a perspective of the user, (ii) determine a distance of the object in the real-world environment relative to the user, (iii) determine a region of the object occluded by the permanently non-transparent display from the user's perspective, (iv) determine a corrected zoom magnitude of the second camera based on the determined region of the object occluded by the permanently non-transparent display and the distance of the object relative to the mobile computing device, and (v) generate, on the permanently non-transparent display of the mobile computing device, a back projection image of the real-world environment captured by the second camera to the permanently non-transparent display based on the corrected zoom magnitude, the determined position of the user's eye relative to the mobile computing device, and at least one device parameter of the mobile computing device such that the back projection image is approximately equivalent to the view of the user of the real world that is occluded by the permanently non-transparent display.

2. The mobile computing device of claim 1 , wherein to generate the back projection comprises to:
determine, for each display pixel of the permanently non-transparent display, a ray from the user's eye through a corresponding display pixel to the object in the real-world environment;
identify, for each determined ray, an image pixel of the second captured image of the real-world environment corresponding with a position of the object in the real-world environment to which the corresponding ray is directed; and
construct a back projection image based on the identified image pixels for display on the permanent non-transparent display of the mobile computing device.

3. The mobile computing device of claim 1 , wherein to determine the corrected zoom magnitude comprises to determine an angular size of a region of the object from a perspective of the second camera corresponding with the region of the object occluded by the mobile computing device from the user's perspective.

4. The mobile computing device of claim 1 , wherein the corrected zoom magnitude is a zoom magnitude required to capture an image with the second camera corresponding with the region of the object occluded by the mobile computing device from the user's perspective.

5. The mobile computing device of claim 4 , wherein the corrected zoom magnitude is a zoom magnitude required to capture an image with the second camera having only image pixels corresponding with features of the object from the region of the object occluded by the mobile computing device from the user's perspective.

6. The mobile computing device of claim 1 , wherein to:
determine the angular size of the mobile computing device from the user's perspective comprises to determine an angular size of the mobile computing device from the user's perspective based on a distance of the user's eye relative to the mobile computing device and a size of the mobile computing device;
determine the distance of the object relative to the user comprises to determine the distance of the object relative to the user based on the distance of the user's eye relative to the mobile computing device and the distance of the object relative to the mobile computing device; and
determine the region of the object occluded by the mobile computing device from the user's perspective comprises to determine the angular size of the region of the object occluded by the mobile computing device based on the angular size of the mobile computing device from the user's perspective and the distance of the object relative to the user.

7. The mobile computing device of claim 1 , wherein angular size, δ, is determined according to

δ
=

2
⁢

⁢

arctan
⁡

(

d

2
⁢
D

)

,

wherein d is an actual size of a corresponding object and D is a distance between the corresponding object and a point, the point being a perspective from which the angular size is determined.

8. The mobile computing device of claim 1 , wherein to:
determine the position of the user's eye relative to the mobile computing device comprises to determine a position of the user's eye relative to the first camera; and
determine the distance of the object in the real-world environment relative to the mobile computing device comprises to determine a distance of the object relative to the second camera.

9. The mobile computing device of claim 1 , wherein the first camera has a field of view in a direction opposite a field of view of the second camera about the permanently non-transparent display.

10. The mobile computing device of claim 1 , wherein to determine the distance of the object in the real-world environment relative to the mobile computing device comprises to set a distance of the object relative to the mobile computing device to a predefined distance.

11. The mobile computing device of claim 1 , further comprising a display module to display an image on the permanently non-transparent display based on the generated back projection of the real-world environment captured by the second camera.

12. The mobile computing device of claim 11 , wherein to display the image based on the generated back projection comprises to display an image corresponding with the back projection modified to include augmented reality features.

13. The mobile computing device of claim 1 , wherein the at least one device parameter comprises at least one of (i) a focal length of the second camera, (ii) a size of the permanently non-transparent display, (iii) a size of the mobile computing device, or (iv) a location of components of the mobile computing device relative to a reference point.

14. One or more non-transitory machine-readable storage media comprising a plurality of instructions stored thereon that, in response to execution by a mobile computing device, cause the mobile computing device to:
capture, by a first camera of the mobile computing device and while the user is holding the mobile computing device, a first image of a user of the mobi

Assignees

• Intel Corp

Inventors

• Reif Dror
• Moran Amit
• Horesh Nizan

Classifications

• G06T2207/30201

  Face · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06F1/163

  Wearable computers, e.g. on a belt · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06T2207/30244

  Camera pose · CPC title