Perception system for autonomous vehicles

Patent metadata

Field	Value
Publication number	US-12100190-B2
Application number	US-202318335785-A
Country	US
Kind code	B2
Filing date	Jun 15, 2023
Priority date	Jun 23, 2020
Publication date	Sep 24, 2024
Grant date	Sep 24, 2024

Abstract

Image processing techniques are described to obtain an image from a camera located on a vehicle while the vehicle is being driven, cropping a portion of the obtained image corresponding to a region of interest, detecting an object in the cropped portion, adding a bounding box around the detected object, determining position(s) of reference point(s) on the bounding box, and determining a location of the detected object in a spatial region where the vehicle is being driven based on the determined one or more positions of the second set of one or more reference points on the bounding box.

First claim

What is claimed is:

1. An image processing method, comprising:
obtaining an image from a camera located on a vehicle;
determining a location of the vehicle when the image is obtained;
selecting a point associated with a region of interest in the image, wherein the point is located at a distance from the location of the vehicle, wherein the point is associated with a three-dimension coordinates that is determined using a map and the location of the vehicle, and wherein the region of interest includes a road or an object on the road;
determining a position of the point in the image, wherein the position of the point in the image is determined by using the three-dimension coordinates associated with the point, camera pose information that describes optical properties of the camera, an orientation of the camera, and a second location of the camera; and
cropping the region of interest in the image based on the position of the point in the image.

2. The method of claim 1 , wherein the obtaining the image, the determining the location, the selecting the point, the determining the position, and the cropping the region of interest are performed as the vehicle is being driven.

3. The method of claim 1 ,
wherein the selecting the point associated with the region of interest in the image includes selecting the point and selecting a second point,
wherein the point is associated with a first distance away from the location of the vehicle,
wherein the second point is associated with a second distance away from the location of the vehicle.

4. The method of claim 3 , wherein the first distance is greater than the second distance.

5. The method of claim 3 , wherein an area of the region of interest is obtained by the position of the point in the image, a second position of the second point in the image, a pre-defined left-most position of the region of interest in the image, and a pre-defined right-most position of the region of interest in the image.

6. The method of claim 5 , wherein a size of the region of interest in the image is determined by the point, the second point, the pre-defined left-most position, and the pre-defined right-most position.

7. A non-transitory computer readable storage medium having code stored thereon, the code, when executed by a processor, causing the processor to implement a method comprising:
obtaining an image from a camera located on a vehicle;
determining a location of the vehicle when the image is obtained;
selecting a point associated with a region of interest in the image, wherein the point is located at a distance from the location of the vehicle, wherein the point is associated with a three-dimension coordinates that is determined using a map and the location of the vehicle, and wherein the region of interest includes a road or an object on the road;
determining a position of the point in the image, wherein the position of the point in the image is determined by using the three-dimension coordinates associated with the point, camera pose information that describes optical properties of the camera, an orientation of the camera, and a second location of the camera; and
cropping the region of interest in the image based on the position of the point in the image.

8. The non-transitory computer readable storage medium of claim 7 , wherein the cropping the region of interest is performed by using the position of the point as a center point of the region of interest.

9. The non-transitory computer readable storage medium of claim 7 , wherein the region of interest has a first resolution that is less than a second resolution of the image.

10. The non-transitory computer readable storage medium of claim 7 , wherein the region of interest is determined by:
determining that the road located in front of the vehicle is straight;
selecting the region of interest in a center of the image in response to the determining that the road is straight.

11. The non-transitory computer readable storage medium of claim 10 , wherein the road is determined to be straight by identifying a curvature or a shape of lane markers associated with the road.

12. The non-transitory computer readable storage medium of claim 7 , wherein the region of interest is determined by:
obtaining information about the road in front of the vehicle from a map;
selecting the region of interest in the image based on the information about the road.

13. The non-transitory computer readable storage medium of claim 7 , wherein the position of the point in the image is determined by determining two-dimension coordinates of a pixel of the point in the image.

14. An apparatus for a vehicle, comprising a processor configured to implement a method, the processor configured to:
obtain an image from a camera located on the vehicle;
determine a location of the vehicle when the image is obtained;
select a point associated with a region of interest in the image, wherein the point is located at a distance from the location of the vehicle, wherein the point is associated with a three-dimension coordinates that is determined using a map and the location of the vehicle, and wherein the region of interest includes a road or an object on the road;
determine a position of the point in the image, wherein the position of the point in the image is determined by using the three-dimension coordinates associated with the point, camera pose information that describes optical properties of the camera, an orientation of the camera, and a second location of the camera; and
crop the region of interest in the image based on the position of the point in the image.

15. The apparatus of claim 14 , wherein the point is located past a pre-determined distance from the location of the vehicle.

16. The apparatus of claim 14 , wherein the point is located within a range of pre-determined distances from the location of the vehicle.

17. The apparatus of claim 14 , wherein the region of interest includes a ramp or an intersection associated with the road.

18. The apparatus of claim 14 , wherein the region of interest in the image is associated with a spatial region towards which the vehicle is driven.

Assignees

• Tusimple Inc

Inventors

• Liu Siyuan
• Ge Lingting
• Qian Chenzhe
• Huang Zehua
• Hou Xiaodi

Classifications

• B60W2420/403

  Image sensing, e.g. optical camera · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06V20/584

  of vehicle lights or traffic lights · CPC title

• B60W60/0025

  specially adapted for specific operations · CPC title