Method and apparatus for depth estimation relative to a mobile terminal

Patent metadata

Field	Value
Publication number	US-12100172-B2
Application number	US-202217671631-A
Country	US
Kind code	B2
Filing date	Feb 15, 2022
Priority date	Feb 22, 2021
Publication date	Sep 24, 2024
Grant date	Sep 24, 2024

Abstract

The method, apparatus and computer program product estimate the depth to objects within a scene that is captured by first and second cameras of a mobile terminal. A method includes determining, based upon an image including an eye of a user captured by a first camera of a mobile terminal and an image of a scene captured by a second camera of the mobile terminal, two or more angles of a geometric shape defined by the eye, the mobile terminal and an object of the scene. Based upon the image captured by the first camera, the method also determines a first distance from the first camera to the eye. The method further estimates the depth to the object in the scene relative to the mobile terminal based upon the first distance and one or more angles of the geometric shape defined by the eye, the mobile terminal and the object.

First claim

The invention claimed is:

1. A method comprising:
based upon an image comprising an eye of a user captured by a first camera of a mobile terminal and an image of a scene captured by a second camera of the mobile terminal, determining two or more angles of a geometric shape defined by the eye of the user, the mobile terminal and an object of the scene;
based upon the image comprising the eye of the user captured by the first camera of the mobile terminal, determining a first distance from the first camera of the mobile terminal to the eye of the user; and
based upon the first distance and one or more angles of the geometric shape defined by the eye of the user, the mobile terminal and the object of the scene, providing for depth estimation to the object in the scene relative to the mobile terminal.

2. The method according to claim 1 , wherein determining two or more angles of the geometric shape comprises:
based upon the image comprising the eye of the user captured by the first camera of the mobile terminal, determining a first angle between a gaze direction of the eye of the user and a path between the mobile terminal and the eye of the user;
based upon the image of the scene captured by the second camera of the mobile terminal and the image comprising the eye of the user captured by the first camera of the mobile terminal, determining a second angle between a path between the mobile terminal and the object in the scene and the path between the mobile terminal and the eye of the user; and
determining a third angle between the path between the mobile terminal and the object in the scene and a path between the eye of the user and the object in the scene.

3. The method according to claim 2 , wherein providing for depth estimation comprises, based upon the first distance, the first angle and the third angle, providing for depth estimation of the object in the scene relative to the mobile terminal.

4. The method according to claim 1 , further comprising causing an instruction to be provided to the user to gaze at the object in the scene prior to the image comprising the eye of the user being captured by the first camera of the mobile terminal.

5. The method according to claim 1 , further comprising receiving information identifying the object in the scene to which a gaze of the user is directed.

6. The method according to claim 1 , further comprising creating a depth map of at least a portion of the scene based at least partially upon the depth estimation of the object in the scene relative to the mobile terminal.

7. The method according to claim 1 , further comprising:
based upon an image comprising a second eye of the user captured by the first camera of the mobile terminal and the image of the scene captured by the second camera of the mobile terminal, determining two or more angles of a geometric shape defined by the second eye of the user, the mobile terminal and the object of the scene; and
based upon the image comprising the second eye of a user captured by the first camera of the mobile terminal, determining a second distance from the first camera of the mobile terminal to the second eye of the user,
wherein providing for depth estimation comprises providing for depth estimation to the object in the scene relative to the mobile terminal also based upon the second distance and one or more angles of the geometric shape defined by the second eye of the user, the mobile terminal and the object of the scene.

8. The method according to claim 1 , further comprising:
receiving information regarding a change in position of the mobile terminal relative to a position of the mobile terminal at a time at which the image comprising the eye of the user was captured by the first camera of the mobile terminal and at a time at which the image of the scene was captured by the second camera of the mobile terminal; and
modifying the depth estimation based upon the change in position of the mobile terminal.

9. An apparatus comprising:
at least one processor; and
at least one memory including computer program code,
the at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to:
based upon an image comprising an eye of a user captured by a first camera of a mobile terminal and an image of a scene captured by a second camera of the mobile terminal, determine two or more angles of a geometric shape defined by the eye of the user, the mobile terminal and an object of the scene;
based upon the image comprising the eye of the user captured by the first camera of the mobile terminal, determine a first distance from the first camera of the mobile terminal to the eye of the user; and
based upon the first distance and one or more angles of the geometric shape defined by the eye of the user, the mobile terminal and the object of the scene, provide for depth estimation to the object in the scene relative to the mobile terminal.

10. The apparatus according to claim 9 , wherein determining two or more angles of the geometric shape comprises:
based upon the image comprising the eye of the user captured by the first camera of the mobile terminal, determining a first angle between a gaze direction of the eye of the user and a path between the mobile terminal and the eye of the user;
based upon the image of the scene captured by the second camera of the mobile terminal and the image comprising the eye of the user captured by the first camera of the mobile terminal, determining a second angle between a path between the mobile terminal and the object in the scene and the path between the mobile terminal and the eye of the user; and
determining a third angle between the path between the mobile terminal and the object in the scene and a path between the eye of the user and the object in the scene.

11. The apparatus according to claim 10 , wherein providing for depth estimation comprises, based upon the first distance, the first angle and the third angle, providing for depth estimation of the object in the scene relative to the mobile terminal.

12. The apparatus according to claim 9 , wherein the at least one memory and the instructions stored therein are configured to, with the at least one processor, further cause the apparatus to: provide an instruction to the user to gaze at the object in the scene prior to the image comprising the eye of the user being captured by the first camera of the mobile terminal.

13. The apparatus according to claim 9 , wherein the at least one memory and the instructions stored therein are configured to, with the at least one processor, further cause the apparatus to: receive information identifying the object in the scene to which a gaze of the user is directed.

14. The apparatus according to claim 9 , wherein the at least one memory and the instructions stored therein are configured to, with the at least one processor, further cause the apparatus to: create a depth map of at least a portion of the scene based at least partially upon the depth estimation of the object in the scene relative to the mobile terminal.

15. The apparatus according to claim 9 , wherein the at least one memory and the instructions stored therein are configured to, with the at least one processor, further cause the apparatus to:
based upon an image comprising a second eye of the user captured by the first camera of the mobile terminal and the image of the scene captured by the second camera of the mobile terminal, determine two or more angles of a geometric shape defined by the second eye of the user, the mobile terminal and the object of the scene; and
based upon the ima

Assignees

• Nokia Technologies Oy

Inventors

• Wright Christopher
• Cronin Harry
• Catton Phil

Classifications

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G01B11/26

  for measuring angles or tapers; for testing the alignment of axes · CPC title

• G01B11/22

  for measuring depth · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title