Depth estimation in images obtained from an autonomous vehicle camera

Patent metadata

Field	Value
Publication number	US-2021398310-A1
Application number	US-202016909987-A
Country	US
Kind code	A1
Filing date	Jun 23, 2020
Priority date	Jun 23, 2020
Publication date	Dec 23, 2021
Grant date	—

Abstract

Image processing techniques are described to receive bounding box information that describes a bounding box located around a detected obj ect in an image, determine one or more positions of one or more reference points on the bounding box, determine, for each reference point, 3D world coordinates of a point of intersection of the reference point and the road surface, and assign the 3D world coordinates of the one or more reference points to a location of the detected object.

First claim

What is claimed is:

1 . An image processing method, comprising:
receiving bounding box information that describes a bounding box located around a detected object in an image, wherein the bounding box information is received while a vehicle is being driven; determining, from the bounding box information and in the image, one or more positions of one or more reference points on the bounding box; for each determined position of each reference point:
determining camera coordinates of a camera center point located on a ray that passes through a position of a reference point, wherein the camera center point is located on a camera coordinate plane located at a focal length distance away from an image plane where the image is received;
determining, based at least on the camera coordinates, first world coordinates of the position of the reference point; and
determining, based on a terrain map, second world coordinates of a point of intersection of the reference point and a road surface, wherein the terrain map provides coordinates of points in a spatial region where the vehicle is being driven; and
assigning the second world coordinates for the one or more reference points to a location of the detected object in the spatial region.

2 . The method of claim 1 , wherein the second world coordinates of the point of intersection is determined by:
obtaining a first set of points along the ray, wherein the reference point belongs to the first set of points; determining a first set of world coordinates corresponding to the first set of points, wherein the first world coordinates belongs to the first set of world coordinates; determining, based on the terrain map and corresponding to the first set of points, a second set of points on the road surface; determining a second set of world coordinates corresponding to the second set of points; determining plurality of heights between each point associated with the first set of world coordinates and a corresponding point associated with the second set of world coordinates; determining a minimum height from the plurality of heights; identifying a point from the second set of points associated with the minimum height; and obtaining world coordinates of the point, wherein the second world coordinates are determined to be same as the world coordinates of the point.

3 . The method of claim 1 , wherein the second world coordinates of the point of intersection is determined by:
determining a first mathematical function that describes the ray; determining, based on the terrain map, world coordinates of two or more points on the bounding box; determining a second mathematical function that describes a plane that includes the two or more points; determining an intersection of the first mathematical function and the second mathematical function; and obtaining world coordinates of the intersection, wherein the second world coordinates are determined to be same as the world coordinates of the intersection.

4 . The method of claim 1 ,
wherein the bounding box includes a plurality of vertices located at a plurality of corners of the bounding box, wherein the one or more reference points includes a reference point located midpoint in between two vertices of the bounding box, and wherein the two vertices are closest to a surface of a road compared to other vertices of the bounding box.

5 . The method of claim 1 , wherein the detected object includes a car, a truck, a truck-trailer, a semi-truck, an emergency vehicle, a pedestrian, a motorcycle, or an obstacle on a road.

6 . The method of claim 1 , wherein the detected object in the image is located at a distance between 500 meters and 1000 meters.

7 . A non-transitory computer readable storage medium having code stored thereon, the code, when executed by a processor, causing the processor to implement a method comprising:
receiving bounding box information that describes a bounding box located around a detected object in an image, wherein the bounding box information is received while a vehicle is being driven; determining, from the bounding box information and in the image, one or more positions of one or more reference points on the bounding box; for each determined position of each reference point:
determining camera coordinates of a camera center point located on a ray that passes through a position of a reference point, wherein the camera center point is located on a camera coordinate plane located at a focal length distance away from an image plane where the image is received;
determining, based at least on the camera coordinates, first world coordinates of the position of the reference point; and
determining, based on a terrain map, second world coordinates of a point of intersection of the reference point and a road surface, wherein the terrain map provides coordinates of points in a spatial region where the vehicle is being driven; and
assigning the second world coordinates for the one or more reference points to a location of the detected object in the spatial region.

8 . The non-transitory computer readable storage medium of claim 7 , wherein a camera intrinsic matrix is used to determine the camera coordinates of the camera center point in the camera coordinate plane.

9 . The non-transitory computer readable storage medium of claim 7 , wherein a camera extrinsic matrix is used with the camera coordinates to determine the first world coordinates of the reference point.

10 . The non-transitory computer readable storage medium of claim 7 , wherein the image is cropped from a second image received from a camera located on the vehicle.

11 . The non-transitory computer readable storage medium of claim 10 , wherein the image is cropped while the vehicle is being driven.

12 . The non-transitory computer readable storage medium of claim 7 , wherein the image is from a first region towards which the vehicle is being driven.

13 . The non-transitory computer readable storage medium of claim 7 , wherein the image is from a second region to a side of the vehicle.

14 . The non-transitory computer readable storage medium of claim 7 , wherein the image is from a third region away from which the vehicle is being driven.

15 . An image processing apparatus for an autonomous vehicle comprising a processor, configured to implement a method comprising:
receiving bounding box information that describes a bounding box located around a detected object in an image, wherein the bounding box information is received while a vehicle is being driven; determining, from the bounding box information and in the image, one or more positions of one or more reference points on the bounding box; for each determined position of each reference point:
determining camera coordinates of a camera center point located on a ray that passes through a position of a reference point, wherein the camera center point is located on a camera coordinate plane located at a focal length distance away from an image plane where the image is received;
determining, based at least on the camera coordinates, first world coordinates of the position of the reference point; and
determining, based on a terrain map, second world coordinates of a point of intersection of the reference point and a road surface, wherein the terrain map provides coordinates of points in a spatial region where the vehicle is being driven; and
assigning the second world coordinates for the one or more reference points to a location of the detected object in the spatial region.

16 . The image processing apparatus of claim 15 , wherein

Assignees

• Tusimple Inc

Inventors

• Ge Lingting
• Liu Siyuan
• Huang Zehua
• Wang Yijie

Classifications

• G06T2207/20104

  Interactive definition of region of interest [ROI] · CPC title

• G06T2207/10012

  Stereo images · CPC title

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title

• G06T7/593 Primary

  from stereo images · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title