Systems and methods for augmented stereoscopic display

Patent metadata

Field	Value
Publication number	US-11106203-B2
Application number	US-201916278042-A
Country	US
Kind code	B2
Filing date	Feb 15, 2019
Priority date	Aug 18, 2016
Publication date	Aug 31, 2021
Grant date	Aug 31, 2021

Abstract

A method for generating a first person view (FPV) of an environment includes, with aid of one or more processors individually or collectively, analyzing stereoscopic video data of the environment to determine environmental information and generating augmented stereoscopic video data of the environment by fusing the stereoscopic video data and the environmental information.

First claim

What is claimed is:

1. A method for generating a first person view (FPV) of an environment, comprising, with aid of one or more processors individually or collectively:
receiving stereoscopic video data of the environment generated by an unmanned aerial vehicle (UAV);
analyzing the stereoscopic video data to determine environmental information; generating an augmented reality (AR) layer that includes the environmental information being configured to assist a navigation of the UAV;
correcting a distortion of images corresponding to the stereoscopic video data based on the environmental information that is obtained through a sensing unit onboard the UAV and includes geometric information about at least one first object within a field of view of an imaging device of the UAV; and
aligning the AR layer to the corrected images to generate, on a terminal remote from the UAV, augmented stereoscopic video data of the environment;
wherein:
the AR layer includes displaying information of the at least one first object and displaying information of at least one second object based on the environmental information obtained through the sensing unit onboard the UAV and includes geometric information about at least one second object outside the field of view of the imaging device of the UAV;
the displaying information of the at least one first object includes at least one of warning information associated with the at least one first object, speed information of the at least one first object, or movement direction information of the at least one first object; and
the displaying information of the at least one second object includes at least one of a proximity level of the at least one second object or a location side of the at least one second object with respect to the UAV.

2. The method of claim 1 , wherein analyzing the stereoscopic video data comprises at least one of stereo matching of image frames or depth map calculation.

3. The method of claim 2 , wherein the stereo matching comprises matching of feature points extracted from the image frames based on motion characteristics of the feature points.

4. The method of claim 2 , wherein the depth map calculation comprises filtering and applying a threshold to a depth map.

5. The method of claim 4 , wherein the threshold is applied to classify objects in the environment having at least one of a predetermined size or a number of pixels in the depth map.

6. The method of claim 1 , wherein the environmental information comprises an environmental map, wherein the environmental map comprises a topological map or a metric map.

7. The method of claim 1 , wherein the environmental information comprises at least one of:
relative positions between the UAV and one or more objects in the environment, or relative positions between two or more objects in the environment.

8. The method of claim 1 , wherein the environmental information comprises at least one of: a distance of the UAV from a third object in the environment, or an orientation of the UAV relative to the third object, the at least one first object including the third object.

9. The method of claim 8 , wherein the UAV is configured to follow or track the third object.

10. The method of claim 8 , wherein the third object is an obstacle lying in the motion path of the UAV, and the UAV is configured to avoid the third object.

11. The method of claim 8 , wherein the augmented stereoscopic video data of the environment comprises a graphical element associated with the environmental information.

12. The method of claim 11 , wherein the graphical element is overlay on the FPV of the environment displayed on the terminal.

13. The method of claim 11 , wherein the graphical element is configured to dynamically change as a position or the orientation of the UAV relative to the third object in the environment changes.

14. The method of claim 12 , wherein the terminal comprises a head-mounted display (HMD).

15. The method of claim 12 , wherein the terminal comprises a pair of virtual reality (VR) or augmented reality (AR) enabled glasses.

16. The method of claim 12 , wherein the terminal comprises a mobile device mounted onto a foldable headgear, the mobile device having a graphical display configured to display the FPV.

17. The method of claim 12 , wherein the terminal allows a user to control and navigate the UAV from the first person view (FPV).

18. The method of claim 1 , further comprising incorporating the environmental information into the stereoscopic video data to fuse the stereoscopic video data and the environmental information.

19. The method of claim 1 , wherein:
the AR layer further includes displaying information of a reference line that extends across the FPV and is relative to a horizontal line of the FPV; and
the method further comprises, in response to a change in a pitch and/or roll of the UAV, obtaining updated environmental information to generate updated AR layer, the updated AR layer including updated displaying information of the reference line indicative of a change in an orientation of the reference line relative to the horizontal line.

20. An apparatus for generating a first person view (FPV) of an environment comprising one or more processors that are individually or collectively configured to:
receive stereoscopic video data of the environment generated by an unmanned aerial vehicle (UAV);
analyze the stereoscopic video data to determine environmental information;
generate an augmented reality (AR) layer that includes the environmental information being configured to assist a navigation of the UAV;
correct a distortion of images corresponding to the stereoscopic video data based on the environmental information that is obtained through a sensing unit onboard the UAV and includes geometric information about at least one first object within a field of view of an imaging device of the UAV; and
align the AR layer to the corrected images to fuse the stereoscopic video data and the environmental information to generate, on a terminal remote from the UAV, augmented stereoscopic video stream of the environment;
wherein:
the AR layer includes displaying information of the at least one first object and displaying information of at least one second object based on the environmental information obtained through the sensing unit onboard the UAV and including geometric information about the at least one second object outside the field of view of the imaging device of the UAV;
the displaying information of the at least one first object includes at least one of warning information associated with the at least one first object, speed information of the at least one first object, or movement direction information of the at least one first object; and
the displaying information of the at least one second object includes at least one of a proximity level of the at least one second object or a location side of the at least one second object with respect to the UAV.

Assignees

• Sz Dji Technology Co Ltd

Inventors

• Huang Zhicong
• Zhao Cong
• Yang Shuo
• Wu Yannan
• Yang Kang
• Zhou Guyue

Classifications

• H04N23/81

  for suppressing or minimising disturbance in the image signal generation · CPC title

• H04N25/61

  the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4" · CPC title

• H04N23/633

  for displaying additional information relating to control or operation of the camera · CPC title

• B64U2201/20

  Remote controls · CPC title

• G06F18/24

  Classification techniques · CPC title