UAV panoramic imaging

Patent metadata

Field	Value
Publication number	US-9854165-B2
Application number	US-201615072220-A
Country	US
Kind code	B2
Filing date	Mar 16, 2016
Priority date	Dec 23, 2014
Publication date	Dec 26, 2017
Grant date	Dec 26, 2017

Abstract

The present invention provides methods, systems, and apparatus for generating panoramic aerial images. An image capturing device is coupled to an unmanned aerial vehicle (UAVs) via a carrier configured to allow the image capturing device to rotate around one, two, or three axes relative to the UAV. To generate a panoramic image, the UAV is brought to hover or remain substantially stationary near a predetermined location. While the UAV is hovering over the predetermined position, the carrier causes the image capturing device to rotate around a first axis while stabilizing the image capturing device with respect to at least a second axis. As the image capturing device rotates, it captures a plurality of overlapping images. The plurality of images can be processed entirely onboard the UAV to generate a panoramic image without transmitting the plurality of images to a remote device.

First claim

What is claimed is:

1. A method for generating panoramic aerial images, comprising:
causing an unmanned aerial vehicle (UAV) carrying an image capturing device onboard the UAV to perform aerial imaging, wherein the image capturing device is coupled to the UAV via a carrier, wherein the UAV comprises a first set of sensors configured to detect at least a position, orientation, or velocity of the UAV, wherein the carrier comprises a second set of sensors configured to detect at least a position, posture, acceleration, or velocity of the carrier or the image capturing device, wherein at least one sensor in the second set of sensors has a higher sampling frequency than at least one sensor in the first set of sensors, and wherein the carrier is configured to permit the image capturing device to rotate around at least two orthogonal axes, the at least two axes comprising a first axis and a second axis;
while the UAV is airborne,
rotating the image capturing device by a predetermined angle of rotation around the first axis that is a yaw axis, and
stabilizing the image capturing device such that the image capturing device substantially maintains a predetermined spatial disposition with respect to at least the second axis that is a roll axis or a pitch axis, wherein said stabilizing the image capturing device comprises controlling the UAV and the carrier collectively, utilizing sensor data acquired by one or more sensors from the first set of sensors to correct sensor data acquired by one or more sensors from the second set of sensors;

capturing, by the image capturing device, a plurality of successive images at predetermined intervals during the stabilized rotation of the image capturing device around the first axis; and
generating by one or more processors onboard the UAV, a panoramic image based on the plurality of images.

2. The method of claim 1 , wherein the predetermined angle of rotation for the image capturing device is at least 360 degrees or less than 360 degrees.

3. The method of claim 1 , wherein the UAV is configured to hover near or at a predetermined location while the image capturing device onboard the UAV is performing the aerial imaging.

4. The method of claim 1 , wherein the second axis is the pitch axis.

5. The method of claim 4 , wherein the carrier is further configured to permit the image capturing device to rotate around a third axis relative to the UAV, and the third axis is the roll axis.

6. The method of claim 1 , wherein the image capturing device is configured to capture the plurality of successive images such that there is at least a 50% overlap between adjacent images of the plurality of images.

7. The method of claim 1 , wherein the generating of the panoramic image comprises processing the plurality of images according to one or more image stitching steps.

8. The method of claim 7 , wherein the at least one of the image stitching steps comprises an image registration step or a global registration step.

9. The method of claim 1 , wherein the generating of the panoramic image overlaps in time with the rotation of the image capturing device around the first axis.

10. The method of claim 1 , wherein the generating of the panoramic image does not overlap in time with the rotation of the image capturing device around the first axis.

11. A system for generating panoramic aerial images, comprising:
a carrier configured to couple with an unmanned aerial vehicle (UAV) and an image capturing device such that the image capturing device is carried onboard the UAV, wherein the carrier is configured to permit the image capturing device to rotate around at least two orthogonal axes relative to the UAV while the image capturing device is performing aerial imaging, the at least two axes comprising a first axis and a second axis;
one or more controllers configured to cause the carrier to, while the UAV is airborne,
rotate the image capturing device by at least 360 degrees around the first axis while the image capturing device captures a plurality of successive images at predetermined intervals, and
stabilize the image capturing device such that the image capturing device substantially maintains a leveled disposition relative to the ground; and

one or more processors onboard the UAV configured to generate a panoramic image based on the plurality of images;
wherein the UAV comprises a first set of sensors configured to detect at least a position, orientation, or velocity of the UAV, wherein the carrier comprises a second set of sensors configured to detect at least a position, posture, acceleration, or velocity of the carrier or the image capturing device, wherein at least one sensor in the second set of sensors has a higher sampling frequency than at least one sensor in the first set of sensors, and
wherein the one or more controllers are configured to stabilize the image capturing device by controlling the UAV and the carrier collectively, utilizing sensor data acquired by one or more sensors from the first set of sensors to correct sensor data acquired by one or more sensors from the second set of sensors.

12. The system of claim 11 , wherein the carrier is configured to stabilize the image capturing device such that the image capturing device substantially maintains a predetermined spatial disposition with respect to at least the second axis during the rotation of the image capturing device.

13. The system of claim 11 , wherein the first axis is a yaw axis and the second axis is a pitch axis.

14. The system of claim 11 , wherein the UAV is configured to hover near or at a predetermined location while the image capturing device onboard the UAV is performing the aerial imaging.

15. The system of claim 11 , wherein the sensor data from at least one of the first and second sets of sensors is used to generate the panoramic image, and wherein the sensor data from the second set of sensors is further used to stabilize the image capturing device.

16. The system of claim 11 , wherein the image capturing device is configured to capture the plurality of successive images such that there is at least a 50% overlap between adjacent images of the plurality of images.

17. The system of claim 11 , wherein the one or more processors onboard the UAV are configured to generate the panoramic image by processing the plurality of images according to one or more image stitching steps, and wherein the one or more image stitching steps comprise image registration, global registration, or blending.

18. The system of claim 17 , wherein a complexity level of at least one of the image stitching steps is reduced when at least one of the following is being performed: omitting an interpolation operation, using a motion model for the plurality of images that does not include rotation, or limiting a scope of a search operation performed under the at least one of the image stitching steps.

19. The system of claim 11 , wherein the generation of the panoramic image by the one or more processors onboard the UAV overlaps in time with the rotation of the image capturing device.

20. The system of claim 11 , wherein the generation of the panoramic image by the one or more processors onboard the UAV does not overlap in time with the rotation of the image capturing device.

21. A system for generating panoramic aerial images, comprising:
a carrier configured to couple with an unmanned aerial vehicle (UAV) and an image capturing device such that the image capturing device is carried onboard the UAV, wherein the carrier is configured

Assignees

• Sz Dji Osmo Technology Co Ltd

Inventors

• Wang Mingyu
• Cao Zisheng
• Liang Taiwen
• Yu Lifu

Classifications

• H04N5/77

  between a recording apparatus and a television camera · CPC title

• H04N23/698 Primary

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/682

  Vibration or motion blur correction · CPC title

• H04N23/685

  performed by mechanical compensation · CPC title

• B64U2101/30

  for imaging, photography or videography · CPC title