Image output adjustment in a robotic vehicle

Patent metadata

Field	Value
Publication number	US-11244468-B2
Application number	US-201716621482-A
Country	US
Kind code	B2
Filing date	Jul 28, 2017
Priority date	Jul 28, 2017
Publication date	Feb 8, 2022
Grant date	Feb 8, 2022

Abstract

Embodiments include devices and methods for processing an image captured by an image sensor of a robotic vehicle. A processor of the robotic vehicle may calculate an image output orientation matrix for an image that is output by the image sensor. The processor may calculate image sensor orientation matrix the image sensor. The processor may calculate a body orientation matrix of the robotic vehicle. The processor may transform the image captured by the image sensor based on the image output orientation matrix, the image sensor orientation matrix, and the body orientation matrix.

First claim

What is claimed is:

1. A method of processing an image captured by an image sensor of a robotic vehicle, comprising:
calculating an image output orientation matrix for an image that is output by the image sensor;
determining an image sensor pose relative to a body orientation of the robotic vehicle;
calculating an image sensor orientation matrix for the image sensor based on the image sensor pose relative to the body orientation of the robotic vehicle;
calculating a body orientation matrix of the robotic vehicle; and
transforming an image captured by the image sensor from an image sensor coordinate system of the image sensor to a body frame coordinate system of the robotic vehicle based on the image output orientation matrix, the image sensor orientation matrix, and the body orientation matrix.

2. The method of claim 1 , wherein transforming the image captured by the image sensor based on the image output orientation matrix, the image sensor orientation matrix, and the body orientation matrix comprises:
calculating a rotation matrix based on the image output orientation matrix, the image sensor orientation matrix, and the body orientation matrix; and
transforming the image captured by the image sensor based on the rotation matrix.

3. The method of claim 1 , wherein the image output orientation matrix is based on an orientation of the image output by the image sensor.

4. The method of claim 1 , wherein the body orientation matrix is based on the body frame coordinate system, and the image sensor orientation matrix is based on the image sensor coordinate system that is different from the body frame coordinate system.

5. The method of claim 1 , wherein the body orientation matrix is based on the body frame coordinate system, the image sensor orientation matrix is based on the image sensor coordinate system that is different from the body frame coordinate system, and the image output orientation matrix is based on a coordinate system that is different from one or both of the body frame coordinate system and the image sensor coordinate system.

6. The method of claim 1 , further comprising determining a relative body orientation of the robotic vehicle, wherein the body orientation matrix is based on the relative body orientation of the robotic vehicle.

7. The method of claim 1 , further comprising outputting the transformed image.

8. A robotic vehicle, comprising:
an image sensor; and
a processor coupled to the image sensor and configured with processor-executable instructions to:
calculate an image output orientation matrix for an image that is output by the image sensor;
determine an image sensor pose relative to a body orientation of the robotic vehicle;
calculate an image sensor orientation matrix for the image sensor based on the image sensor pose relative to the body orientation of the robotic vehicle;
calculate a body orientation matrix of the robotic vehicle; and
transform an image captured by the image sensor from an image sensor coordinate system of the image sensor to a body frame coordinate system of the robotic vehicle based on the image output orientation matrix, the image sensor orientation matrix, and the body orientation matrix.

9. The robotic vehicle of claim 8 , wherein the processor is further configured to:
calculate a rotation matrix based on the image output orientation matrix, the image sensor orientation matrix, and the body orientation matrix; and
transform the image captured by the image sensor based on the rotation matrix.

10. The robotic vehicle of claim 8 , wherein the processor is further configured such that the image output orientation matrix is based on an orientation of the image output by the image sensor.

11. The robotic vehicle of claim 8 , wherein the processor is further configured such that the body orientation matrix is based on the body frame coordinate system, and the image sensor orientation matrix is based on the image sensor coordinate system that is different from the body frame coordinate system.

12. The robotic vehicle of claim 8 , wherein the processor is further configured such that the body orientation matrix is based on the body frame coordinate system, the image sensor orientation matrix is based on the image sensor coordinate system that is different from the body frame coordinate system, and the image output orientation matrix is based on a coordinate system that is different from one or both of the body frame coordinate system and the image sensor coordinate system.

13. The robotic vehicle of claim 8 , wherein the processor is further configured to:
determine a relative body orientation of the robotic vehicle, wherein the body orientation matrix is based on the relative body orientation of the robotic vehicle.

14. The robotic vehicle of claim 8 , wherein the processor is further configured to:
output the transformed image.

15. A processing device for use in a robotic vehicle configured to:
calculate an image output orientation matrix for an image that is output by an image sensor;
determine an image sensor pose relative to a body orientation of the robotic vehicle;
calculate an image sensor orientation matrix for the image sensor based on the image sensor pose relative to the body orientation of the robotic vehicle;
calculate a body orientation matrix of the robotic vehicle; and
transform an image captured by the image sensor from an image sensor coordinate system of the image sensor to a body frame coordinate system of the robotic vehicle based on the image output orientation matrix, the image sensor orientation matrix, and the body orientation matrix.

16. The processing device of claim 15 , further configured to:
calculate a rotation matrix based on the image output orientation matrix, the image sensor orientation matrix, and the body orientation matrix; and
transform the image captured by the image sensor based on the rotation matrix.

17. The processing device of claim 15 , further configured such that the image output orientation matrix is based on an orientation of the image output by the image sensor.

18. The processing device of claim 15 , further configured such that the body orientation matrix is based on the body frame coordinate system, and the image sensor orientation matrix is based on the image sensor coordinate system that is different from the body frame coordinate system.

19. The processing device of claim 15 , further configured such that the body orientation matrix is based on the body frame coordinate system, the image sensor orientation matrix is based on the image sensor coordinate system that is different from the body frame coordinate system, and the image output orientation matrix is based on a coordinate system that is different from one or both of the body frame coordinate system and the image sensor coordinate system.

20. The processing device of claim 15 , further configured to:
determine a relative body orientation of the robotic vehicle, wherein the body orientation matrix is based on the relative body orientation of the robotic vehicle.

21. The processing device of claim 15 , further configured to output the transformed image.

22. A robotic vehicle, comprising:
an image sensor;
means for calculating an image output orientation matrix for an image that is output by the image sensor;
means for determining an image sensor pose relative to a body orientation of the robotic vehicle;
means for calculating an image sensor orientation matrix for the image sensor

Assignees

• Qualcomm Inc

Inventors

• Huang Yin
• Zhu Xiaoyi
• Chan Leslie Chiming
• Wang Ruowei

Classifications

• G06T7/30

  Determination of transform parameters for the alignment of images, i.e. image registration · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T7/70 Primary

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G01C21/20 Primary

  Instruments for performing navigational calculations (G01C21/24, G01C21/26 take precedence) · CPC title

• G06T2207/10032

  Satellite or aerial image; Remote sensing · CPC title