User interaction paradigms for a flying digital assistant

Patent metadata

Field	Value
Publication number	US-10795353-B2
Application number	US-201916581020-A
Country	US
Kind code	B2
Filing date	Sep 24, 2019
Priority date	Jun 19, 2014
Publication date	Oct 6, 2020
Grant date	Oct 6, 2020

Abstract

Methods and systems are described for new paradigms for user interaction with an unmanned aerial vehicle (referred to as a flying digital assistant or FDA) using a portable multifunction device (PMD) such as smart phone. In some embodiments, a user may control image capture from an FDA by adjusting the position and orientation of a PMD. In other embodiments, a user may input a touch gesture via a touch display of a PMD that corresponds with a flight path to be autonomously flown by the FDA.

First claim

What is claimed is:

1. A method comprising:
receiving sensor data gathered by sensors onboard an unmanned aerial vehicle (UAV) in flight through a physical environment;
generating a three-dimensional (3D) model of the physical environment by processing the sensor data received from the UAV;
determining a position and orientation of a display device in the physical environment;
placing a virtual camera at a position and orientation in the 3D model of the physical environment that corresponds to the determined position and orientation of the display device in the physical environment;
generate a visual representation of at least a portion of the 3D model of the physical environment from a field of view of the virtual camera; and
causing display, via the display device, of the generated visual representation.

2. The method of claim 1 , wherein the display device is any of a virtual reality display device or augmented reality display device.

3. The method of claim 1 , wherein the sensors onboard the UAV include any one or more of: an optical sensor, a motion sensor, or a proximity sensor.

4. The method of claim 1 , wherein the 3D model of the physical environment is generated and continually updated in real time or near real time as the UAV is in flight through the physical environment.

5. The method of claim 1 , wherein the generated visual representation of the 3D model includes a visual representation of a physical object in the physical environment that is not in a line of sight with the display device.

6. The method of claim 1 , further comprising:
tracking a motion of a physical object through the physical environment by processing the sensor data gathered by the sensors onboard the UAV; and
updating the generated 3D model of the physical environment based on the tracked motion of the physical object through the physical environment.

7. The method of claim 1 , wherein the generated visual representation includes a graphical user interaction element, the method further comprising:
detecting a user interaction with the graphical user interaction element; and
generating control commands configured to cause the UAV to maneuver based on the detected user interaction.

8. The method of claim 1 , further comprising:
detecting a change in the position and/or orientation of the display device in the physical environment; and
generating control commands configured to cause the UAV to maneuver based on the detected change in position and/or orientation of the display device in the physical environment.

9. A computer system comprising:
a processor; and
a memory coupled to the processor, the memory having instructions stored thereon, which when executed by the processor cause the computer system to:
receive sensor data gathered by sensors onboard an unmanned aerial vehicle (UAV) in flight through a physical environment;
generate a three-dimensional (3D) model of the physical environment by processing the sensor data received from the UAV;
determine a position and orientation of a display device in the physical environment;
place a virtual camera at a position and orientation in the 3D model of the physical environment that corresponds to the determined position and orientation of the display device in the physical environment;
generate a visual representation of at least a portion of the 3D model of the physical environment from a field of view of the virtual camera; and
cause display, via the display device, of the generated visual representation.

10. The computer system of claim 9 , wherein the 3D model of the physical environment is generated and continually updated in real time or near real time as the UAV is in flight through the physical environment.

11. The computer system of claim 9 , wherein the generated visual representation of the 3D model includes a visual representation of a physical object in the physical environment that is not in a line of sight with the display device.

12. The computer system of claim 9 , wherein the memory has further instructions stored thereon, which when executed by the processor, cause the computer system to further:
track a motion of a physical object through the physical environment by processing the sensor data gathered by the sensors onboard the UAV; and
update the generated 3D model of the physical environment based on the tracked motion of the physical object through the physical environment.

13. The computer system of claim 9 , wherein the generated visual representation includes a graphical user interaction element, wherein the memory has further instructions stored thereon, which when executed by the processor, cause the computer system to further:
detect a user interaction with the graphical user interaction element; and
generate control commands configured to cause the UAV to maneuver based on the detected user interaction.

14. The computer system of claim 9 , wherein the memory has further instructions stored thereon, which when executed by the processor, cause the computer system to further:
detect a change in the position and/or orientation of the display device in the physical environment; and
generate control commands configured to cause the UAV to maneuver based on the detected change in position and/or orientation of the display device in the physical environment.

15. A mobile device comprising:
a display device;
an accelerometer;
a global positioning system (GPS) receiver;
a wireless communication interface; and
a computer system configured to:
access, via the wireless communication interface, a three-dimensional (3D) model of a physical environment, the 3D model of the physical environment generated based on sensor data from an unmanned aerial vehicle (UAV) in flight through the physical environment
determine, based on data from the accelerometer and GPS receiver, a position and orientation of the mobile device in the physical environment;
place a virtual camera at a position and orientation in the 3D model of the physical environment that corresponds to the determined position and orientation of the mobile device in the physical environment;
generate a visual representation of at least a portion of the 3D model of the physical environment from a field of view of the virtual camera; and
cause display, via the display device, of the generated visual representation.

16. The mobile device of claim 15 , wherein the 3D model of the physical environment is generated and continually updated in real time or near real time as the UAV is in flight through the physical environment.

17. The mobile device of claim 15 , wherein the generated visual representation of the 3D model includes a visual representation of a physical object in the physical environment that is not in a line of sight with the mobile device.

18. The mobile device of claim 15 , wherein the display device is any of a virtual reality display device or augmented reality display device.

19. The mobile device of claim 15 , wherein the generated visual representation includes a graphical user interaction element, wherein the computer system is configured to further:
detect a user interaction with the graphical user interaction element; and
transmit, via the wireless communication interface, a command signal to the UAV, the configured to cause the UAV to autonomously maneuver based on the detected user interaction.

20. The mobile device of claim 15 , wherein the computer system is configured to further:
detect, based on updated data from the from the accelerome

Assignees

• Skydio Inc

Inventors

• Bachrach Abraham Galton
• Bry Adam Parker
• Donahoe Matthew Joseph

Classifications

• B64U2201/10

  autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS] · CPC title

• B64U2201/20

  Remote controls · CPC title

• H04N23/695

  Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects · CPC title

• H04N23/661

  Transmitting camera control signals through networks, e.g. control via the Internet · CPC title

• B64U2101/30

  for imaging, photography or videography · CPC title