Depth estimation in images obtained from an autonomous vehicle camera

Patent metadata

Field	Value
Publication number	US-11461922-B2
Application number	US-202016909987-A
Country	US
Kind code	B2
Filing date	Jun 23, 2020
Priority date	Jun 23, 2020
Publication date	Oct 4, 2022
Grant date	Oct 4, 2022

Abstract

Image processing techniques are described to receive bounding box information that describes a bounding box located around a detected object in an image, determine one or more positions of one or more reference points on the bounding box, determine, for each reference point, 3D world coordinates of a point of intersection of the reference point and the road surface, and assign the 3D world coordinates of the one or more reference points to a location of the detected object.

First claim

What is claimed is:

1. An image processing method, comprising:
receiving bounding box information that describes a bounding box located around a detected object in an image, wherein the bounding box information is received while a vehicle is being driven;
determining, from the bounding box information and in the image, one or more positions of one or more reference points on the bounding box;
for each determined position of each reference point:
determining camera coordinates of a camera center point that is located on a camera coordinate plane located at a focal length distance away from an image plane where the image is received;
determining, based at least on the camera coordinates and the focal length distance, first world coordinates of a position of a reference point; and
determining, based on a terrain map and using a ray that passes through the camera coordinates of the camera center point and the first world coordinates of the position of the reference point in the image plane, second world coordinates of a point of intersection where the ray comprising the reference point intersects a road surface, wherein the terrain map provides coordinates of points in a spatial region where the vehicle is being driven; and

assigning the second world coordinates for the one or more reference points to a location of the detected object in the spatial region.

2. The method of claim 1 , wherein the second world coordinates of the point of intersection are determined by:
obtaining a first set of points along the ray, wherein the reference point belongs to the first set of points;
determining a first set of world coordinates corresponding to the first set of points, wherein the first world coordinates belong to the first set of world coordinates;
determining, based on the terrain map and corresponding to the first set of points, a second set of points on the road surface;
determining a second set of world coordinates corresponding to the second set of points;
determining plurality of heights between each point associated with the first set of world coordinates and a corresponding point associated with the second set of world coordinates;
determining a minimum height from the plurality of heights;
identifying a point from the second set of points associated with the minimum height; and obtaining world coordinates of the point, wherein the second world coordinates are determined to be same as the world coordinates of the point.

3. The method of claim 1 , wherein the second world coordinates of the point of intersection are determined by:
determining a first mathematical function that describes the ray;
determining, based on the terrain map, world coordinates of two or more points on the bounding box;
determining a second mathematical function that describes a plane that includes the two or more points;
determining an intersection of the first mathematical function and the second mathematical function; and
obtaining world coordinates of the intersection, wherein the second world coordinates are determined to be same as the world coordinates of the intersection.

4. The method of claim 1 , wherein the bounding box includes a plurality of vertices located at a plurality of corners of the bounding box, wherein the one or more reference points includes a reference point located midpoint in between two vertices of the bounding box, and wherein the two vertices are closest to a surface of a road compared to other vertices of the bounding box.

5. The method of claim 1 , wherein the detected object includes a car, a truck, a truck-trailer, a semi-truck, an emergency vehicle, a pedestrian, a motorcycle, or an obstacle on a road.

6. The method of claim 1 , wherein the detected object in the image is located at a distance between 500 meters and 1000 meters.

7. A non-transitory computer readable storage medium having code stored thereon, the code, when executed by a processor, causing the processor to implement a method comprising:
receiving bounding box information that describes a bounding box located around a detected object in an image, wherein the bounding box information is received while a vehicle is being driven;
determining, from the bounding box information and in the image, one or more positions of one or more reference points on the bounding box;
for each determined position of each reference point:
determining camera coordinates of a camera center point that is located on a camera coordinate plane located at a focal length distance away from an image plane where the image is received;
determining, based at least on the camera coordinates and the focal length distance, first world coordinates of the position of the reference point; and
determining, based on a terrain map and using a ray that passes through the camera coordinates of the camera center point and the first world coordinates of the position of the reference point in the image plane, second world coordinates of a point of intersection where the ray comprising the reference point intersects a road surface, wherein the terrain map provides coordinates of points in a spatial region where the vehicle is being driven; and

assigning the second world coordinates for the one or more reference points to a location of the detected object in the spatial region.

8. The non-transitory computer readable storage medium of claim 7 , wherein a camera intrinsic matrix is used to determine the camera coordinates of the camera center point in the camera coordinate plane.

9. The non-transitory computer readable storage medium of claim 7 , wherein a camera extrinsic matrix is used with the camera coordinates to determine the first world coordinates of the reference point.

10. The non-transitory computer readable storage medium of claim 7 , wherein the image is cropped from a second image received from a camera located on the vehicle.

11. The non-transitory computer readable storage medium of claim 10 , wherein the image is cropped while the vehicle is being driven.

12. The non-transitory computer readable storage medium of claim 7 , wherein the image is from a first region towards which the vehicle is being driven.

13. The non-transitory computer readable storage medium of claim 7 , wherein the image is from a second region to a side of the vehicle.

14. The non-transitory computer readable storage medium of claim 7 , wherein the image is from a third region away from which the vehicle is being driven.

15. An image processing apparatus for an autonomous vehicle comprising at least one processor; and
at least one memory including computer program code which, when executed by the at least one processor, cause the image processing apparatus to at least:
receive bounding box information that describes a bounding box located around a detected object in an image, wherein the bounding box information is received while a vehicle is being driven;
determine, from the bounding box information and in the image, one or more positions of one or more reference points on the bounding box;
for each determined position of each reference point:
determine camera coordinates of a camera center point that is located on a camera coordinate plane located at a focal length distance away from an image plane where the image is received;
determine, based at least on the camera coordinates and the focal length distance, first world coordinates of the position of the reference point; and
determine, based on a terrain map and using a ray that passes through the camera coordinates of the camera center point and the first world coordinates of the position of the reference point in the i

Assignees

• Tusimple Inc

Inventors

• Ge Lingting
• Liu Siyuan
• Huang Zehua
• Wang Yijie

Classifications

• G06V20/58 Primary

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06T2207/30261

  Obstacle · CPC title

• G06T7/593 Primary

  from stereo images · CPC title

• G01C21/30

  Map- or contour-matching · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title