Method of multi-drone camera control

Patent metadata

Field	Value
Publication number	US-11256257-B2
Application number	US-202016917671-A
Country	US
Kind code	B2
Filing date	Jun 30, 2020
Priority date	Jun 30, 2020
Publication date	Feb 22, 2022
Grant date	Feb 22, 2022

Abstract

A method of camera control for a camera capturing images of a target includes a sequence of at least four steps. The first step determines in real time a location of a drone and a pose of a camera on the drone. The second step, which may occur before or after the first step, determines in real time a location of a reference object, the location of the reference object having a fixed relationship to a location of the target. The third step uses the determined locations to calculate a distance, characterized by magnitude and direction, between the target and the drone. The fourth step uses the calculated distance to control the pose of the camera such that an image captured by the camera includes the target. Controlling the pose of the camera does not require any analysis of the captured image.

First claim

We claim:

1. A method of automatic camera control for a camera capturing images of a target, the method comprising:
a sequence of at least four steps, the sequence comprising:
in a first step, using a first technique to determine in real time a location of a drone and a pose of a camera on the drone;
in a second step, which may occur before or after the first step, using a second technique to determine in real time a location of a reference object, the location of the reference object having a fixed relationship to a location of the target;
in a third step, using the determined locations to calculate a distance, characterized by magnitude and direction, between the reference object and the drone; and
in a fourth step, using the calculated distance to control the pose of the camera such that an image captured by the camera includes the target;
wherein controlling the pose of the camera does not require any analysis of the captured image; and
wherein no human participation or intervention is required during execution of the sequence.

2. The method of claim 1 , wherein the fourth step further comprises optimizing image quality by controlling one or more image capture parameters of the camera, adjusting at least one image capture parameter on the basis of the calculated distance, the controlled image capture parameters including at least one of focal length, aperture, shutter speed, and zoom.

3. The method of claim 1 ,
wherein the sequence additionally comprises a fifth step after the fourth step, the fifth step comprising moving the drone to a position close to an anticipated next location of the target; and
wherein the sequence is carried out repeatedly and sufficiently quickly to enable real time capture by the camera of a series of images tracking the target in motion.

4. The method of claim 1 ,
wherein the first technique is an RTK-GNSS-IMU technique; and
wherein the second technique comprises one of an RTK-GNSS-IMU technique, LiDAR, use of a stereo camera, use of an RGBD camera, and multi-view triangulation.

5. The method of claim 1 , wherein determining the location and pose of the drone comprises data exchange between a Ground Control Station and a drone controller in the drone.

6. The method of claim 1 , wherein controlling the pose of the camera comprises using a 3D gimbal controlled by one of: an integrated drone controller in the drone, a dedicated camera controller in the drone, and an external camera controlling station.

7. A method of automatic camera control for a plurality of cameras capturing images of a target, the method comprising:
a sequence of at least four steps, the sequence comprising:
in a first step, using a first technique to determine in real time, for each drone in a plurality of drones, a location of that drone and a pose of a camera on that drone;
in a second step, which may occur before or after the first step, using a second technique to determine a location of a reference object, the location of the reference object having a fixed relationship to a location of the target;
in a third step, for each one of the plurality of drones, using the determined locations to calculate a distance, characterized by magnitude and direction, between the reference object and that drone; and
in a fourth step, for each one of the plurality of drones, using the corresponding calculated distance to control the pose of the corresponding camera such that an image captured by that camera includes the target;

wherein control of the pose of each camera does not require any analysis of the corresponding captured image; and
wherein no human participation or intervention is required during execution of the sequence.

8. The method of claim 7 , wherein the fourth step further comprises optimizing image quality by controlling one or more image capture parameters of the camera, adjusting at least one image capture parameter on the basis of the calculated distance, the controlled image capture parameters including at least one of focal length, aperture, shutter speed, and zoom.

9. The method of claim 7 ,
wherein the sequence comprises a fifth step after the fourth step, the fifth step comprising, for each drone in the plurality of drones, moving the drone to a position close to an anticipated next location of the target; and
wherein the sequence is carried out repeatedly and sufficiently quickly to enable real time capture by the camera of a series of images tracking the target in motion.

10. The method of claim 9 wherein the first technique is an the RTK-GNSS-IMU technique, and the second technique is one of an RTK-GNSS-IMU technique, LiDAR, stereo cameras, an RGBD camera, and multi-view triangulation.

11. The method of claim 9 , wherein moving each drone to a position close to an anticipated next location of the moving target does not depend on calculating the anticipated next location based on any predetermined plan of motion for the target.

12. The method of claim 9 , wherein moving each drone to a position close to an anticipated next location of the moving target is carried out in part under supervision by a human operator.

13. The method of claim 7 , wherein determining the location and pose of each drone comprises data exchange between a Ground Control Station, shared by the plurality of drones, and a drone controller in each drone.

14. The method of claim 7 ,
wherein one of the drones in the plurality of drones is a Master Drone, having a controller in direct or indirect communication with each other drone in the plurality; and
wherein determining the location and pose of each drone comprises data exchange between a Ground Control Station (GCS) and the controller in the Master drone, no communication occurring between the GCS and any other drone in the plurality except for the Master drone.

15. The method of claim 7 , wherein the plurality of drones is distributed in space according to a pattern that changes over time, changes in the pattern occurring under control of at least one of: a Ground Control Station communicating directly with each drone, and a distributed control system comprising communication between controllers on different ones of the drones in the plurality of drones.

16. The method of claim 7 , wherein controlling the pose of each camera comprises using a 3D gimbal controlled by one of: an integrated drone controller in the corresponding drone, a dedicated camera controller in the corresponding drone, and a camera controlling station shared by all drones in the plurality of drones.

17. A system of automatic camera control for a plurality of cameras capturing images of a target, the system comprising:
one or more processors; and
logic encoded in one or more non-transitory media for execution by the one or more processors and when executed operable to capture images of a target by carrying out a sequence of at least four steps, the sequence comprising:
in a first step, using a first technique to determine in real time, for each drone in a plurality of drones, a location of that drone and a pose of a camera on that drone;
in a second step, which may occur before or after the first step, using a second technique to determine a location of a reference object, the location of the reference object having a fixed relationship to a location of the target;
in a third step, for each one of the plurality of drones, using the determined locations to calculate a distance, characterized by magnitude and direction, between the target and that drone; and
in a fourth step, for each one of the plurality of drones, using the corresponding calc

Assignees

• Sony Group Corp

Inventors

• Berestov Alexander
• Liu Cheng-Yi
• Chen Jianfei

Classifications

• H04N25/46

  by combining or binning pixels · CPC title

• H04N23/695

  Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects · CPC title

• H04N23/62

  Control of parameters via user interfaces · CPC title

• H04N23/69

  Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming · CPC title

• H04N23/60

  Control of cameras or camera modules · CPC title