User interface to facilitate control of unmanned aerial vehicles (UAVs)

Patent metadata

Field	Value
Publication number	US-9927809-B1
Application number	US-201514928310-A
Country	US
Kind code	B1
Filing date	Oct 30, 2015
Priority date	Oct 31, 2014
Publication date	Mar 27, 2018
Grant date	Mar 27, 2018

Abstract

Various techniques are described to facilitate the control of an unmanned aerial vehicle (UAV). A graphical user interface (GUI) is provided that allows a user to control a UAV using familiar gestures. The GUI may include live video that is captured and transmitted by a UAV, which may be displayed in a display window on a portable computing device. Upon a user interacting with the display window, the video may be updated to display a new UAV perspective by changing a zoom level, panning, rotating, etc., based upon the type of user interaction. In response to the user interaction, a command may be generated and transmitted to the UAV, causing the UAV to navigate to a location to reflect the change in perspective. Upon the UAV moving to the new position, the live video data that is displayed may match the perspective indicated by the user gesture.

First claim

What is claimed is:

1. A computer-implemented method for controlling an unmanned aerial vehicle (UAV), comprising:
displaying, by one or more processors, live video received from a UAV overlaid onto a map image displayed within a display window;
receiving, by one or more processors, a first user input to update the overlaid position of the live video within the map image in response to a user interacting with the map image displayed in the display window;
adjusting, by one or more processors, the live video in response to the first user input to update the overlaid position of the live video within the map image displayed within the display window;
translating, by one or more processors, a change in the overlaid position of the live video with respect to the map image as an equivalent change in the UAV's mapped position;
generating, by one or more processors, one or more UAV commands based upon the equivalent change in the UAV's mapped position using the translated change in the-overlaid position of the live video with respect to the map image; and
transmitting, by one or more processors, the one or more UAV commands to the UAV.

2. The computer-implemented method of claim 1 , wherein the first user input includes
a gesture to pan the overlaid position of the live video displayed within the display window.

3. The computer-implemented method of claim 1 , wherein:
the first user input includes a user panning the overlaid position of the live video by a panning distance,
the act of translating the change in the position of the overlaid position of the live video with respect to the map image comprises:
translating the panning distance as a change in UAV geographical position, and the act of generating the one or more UAV commands comprises:
generating the one or more UAV commands to cause the UAV to change its geographical position as a result of the live video being panned by the panning distance.

4. The computer-implemented method of claim 1 , further comprising:
receiving, by one or more processors, a second user input corresponding to the user changing a zoom level of the live video displayed in the display window.

5. The computer-implemented method of claim 4 , further comprising:
translating the change in zoom level as a change in UAV altitude; and
generating one or more UAV commands to cause the UAV to change its altitude as a result of the change in zoom level.

6. The computer-implemented method of claim 1 , further comprising:
receiving, by one or more processors, a third user input corresponding to the user selecting an object within the display window; and
generating, by one or more processors, one or more UAV commands to cause the UAV to perform an image analysis of the object based upon the third user input.

7. A portable computing device, comprising:
an interactive display configured to display live video received from a UAV overlaid onto a map image displayed within a display window;
a user interface configured to receive first user input to update the overlaid position of the live video within the map image in response to a user interacting with the map image displayed in the display window;
a graphical processing unit (GPU) configured to adjust the live video in response to the user input to update the overlaid position of the live video within the map image displayed within the display window;
a processor configured to translate a change in the overlaid position of the live video with respect to the map image as an equivalent change in the UAV's mapped position and to generate one or more UAV commands based upon the equivalent change in the UAV's position using the translated change in the overlaid position of the live video with respect to the map image; and
a communication unit configured to transmit the one or more UAV commands to the UAV.

8. The portable computing device of claim 7 , wherein the first user input includes a gesture to pan the overlaid position of the live video displayed in the display window.

9. The portable computing device of claim 7 , wherein:
the first user input includes a user panning the overlaid position of the live video by a panning distance, and
the processor is further configured to translate the change in the overlaid position of the live video with respect to the map image by translating the panning distance as a change in UAV geographical position, and to generate the one or more UAV commands to cause the UAV to change its geographical position as a result of the live video being panned by the panning distance.

10. The portable computing device of claim 7 , wherein the user interface is further configured to receive a second user input corresponding to the user changing a zoom level of the live video feed displayed in the display window.

11. The portable computing device of claim 10 , wherein the processor is further configured to translate the change in zoom level as a change in UAV altitude, and to generate the one or more UAV commands to cause the UAV to change its altitude as a result of the change in zoom level.

12. The computer-implemented method of claim 1 , wherein the user interface is further configured to receive a third user input in response to the user selecting an object within the display window, and wherein the processor is further configured to generate the one or more UAV commands to cause the UAV to perform an image analysis of the object based upon the third user input.

13. A computer-implemented method for controlling an unmanned aerial vehicle (UAV), comprising:
displaying, by one or more processors, live video received from a UAV in a first display window;
displaying, by one or more processors, a map image in a second display window that includes the first display window, the first display window being superimposed over the second display window at a location on the map image that corresponds to the geographic location of the UAV while the live video is being captured;
receiving, by one or more processors a first user input to update the live video in the first display window in response to a user interacting with the first display window;
receiving, by one or more processors, a second user input corresponding to a user panning the map image displayed in the second display window to move the superimposed position of the first display window with respect to the map image;
adjusting, by one or more processors, the live video in response to the first user input to update the live video in the first display window;
translating, by one or more processors, (i) a change in the UAV's perspective resulting from the live video being adjusted as an equivalent change in the UAV's altitude, and (ii) a change in the superimposed position of the first display window with respect to the map image as an equivalent change in the UAV's mapped position;
generating, by one or more processors, one or more UAV commands based upon the change in the UAV's altitude and mapped position; and
transmitting, by one or more processors, the one or more UAV commands to the UAV.

14. The computer-implemented method of claim 13 , wherein:
the first display window maintains a superimposed position relative to the edges of the second display window while the second display window is panned such that the superimposed position of the first display window is moved to an updated map position within the map image included in the second display window, and
the one or more UAV commands cause the UAV to move to a geographical position that corresponds to the updated mapped position within the map image.

15. The computer-implemented method of clai

Assignees

• State Farm Mutual Automobile Insurance Co

Inventors

• Tofte Nathan L
• Harvey Brian N

Classifications

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N23/661

  Transmitting camera control signals through networks, e.g. control via the Internet · CPC title

• H04N23/632

  for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters · CPC title

• B64U2201/20

  Remote controls · CPC title

• G06F3/04847

  Interaction techniques to control parameter settings, e.g. interaction with sliders or dials · CPC title