Partitioning images obtained from an autonomous vehicle camera

Patent metadata

Field	Value
Publication number	US-11373389-B2
Application number	US-202016909962-A
Country	US
Kind code	B2
Filing date	Jun 23, 2020
Priority date	Jun 23, 2020
Publication date	Jun 28, 2022
Grant date	Jun 28, 2022

Abstract

Image processing techniques are described to select and crop a region of interest from an image obtained from a camera located on or in a vehicle, such as an autonomous semi-trailer truck. The region of interest can be identified by selecting one or more reference points and determining one or more positions of the one or more reference points on the image obtained from the camera. As an example, a location of two reference points may be 500 meters and 1000 meters in front of a location of autonomous vehicle, where the front of the autonomous vehicle is an area towards which the autonomous vehicle is being driven.

First claim

What is claimed is:

1. An image processing method, comprising:
obtaining an image from a camera located on a vehicle while the vehicle is being driven, wherein the image is associated with a time that describes when the image was obtained;
determining a location of the vehicle in a spatial region where the vehicle is being driven, wherein the location is associated with the time when the image was obtained;
determining presence of a plurality of region of interest (ROI) relative to the location of the vehicle;
determining, in the image, locations of one or more ROI from the plurality of ROI;
determining, after the determining the locations of the one or more ROI in the image, that the image includes, from the one or more ROI, a second ROI that is same as that selected or cropped in a second image that is obtained at a second time prior to the time when the image is obtained; and
cropping a portion of the image associated with a first ROI from the one or more ROI, wherein the second ROI is determined to be excluded from the one or more ROI prior to the cropping the portion of the image associated with the first ROI.

2. The method of claim 1 , wherein the one or more ROI are selected from the plurality of ROI in response to determining that the one or more ROI affect or have a potential to affect an operation of the vehicle.

3. The method of claim 1 , wherein the second image is obtained by another camera located on the vehicle.

4. The method of claim 1 , wherein the presence of the plurality of ROI is determined by querying a database and obtaining three-dimensional world coordinates of the plurality of ROIs that are located within a pre-determined distance from the location of the vehicle.

5. The method of claim 4 , wherein the location of the plurality of ROI in the image is determined by projecting the three-dimensional world coordinates of the plurality of ROI to the image.

6. The method of claim 4 , wherein the pre-determined distance includes 500 meters, 750 meters, or 1000 meters.

7. The method of claim 2 , wherein the plurality of ROI include a highway on-ramp, a traffic light, a stop sign, road lanes, road sign, or road markers, and wherein the one or more ROI include the highway on-ramp, the traffic light, or the stop sign.

8. A non-transitory computer readable storage medium having code stored thereon, the code, when executed by a processor, causing the processor to implement a method comprising:
obtaining an image from a camera located on a vehicle while the vehicle is being driven, wherein the image is associated with a time that describes when the image was obtained;
determining a location of the vehicle in a spatial region where the vehicle is being driven, wherein the location is associated with the time when the image was obtained;
determining presence of a plurality of region of interest (ROI) relative to the location of the vehicle;
determining, in the image, locations of one or more ROI from the plurality of ROI;
determining, after the determining the locations of the one or more ROI in the image, that the image includes, from the one or more ROI, a second ROI that is same as that selected or cropped in a second image that is obtained at a second time prior to the time when the image is obtained; and
cropping a portion of the image associated with a first ROI from the one or more ROI, wherein the second ROI is determined to be excluded from the one or more ROI prior to the cropping the portion of the image associated with the first ROI.

9. The non-transitory computer readable storage medium of claim 8 , wherein prior to the determining the locations of the one or more ROI, the method comprises:
selecting the one or more ROI from the plurality of ROI in response to determining that the one or more ROI affect or have a potential to affect an operation of the vehicle.

10. The non-transitory computer readable storage medium of claim 8 , wherein the presence of the plurality of ROI is determined by querying a database and obtaining three-dimensional world coordinates of the plurality of ROIs that are located within a pre-determined distance from the location of the vehicle.

11. The non-transitory computer readable storage medium of claim 8 , wherein the method further comprises:
selecting one or more reference points located at one or more pre-determined distances relative to the location of the vehicle;
determining one or more positions of the one or more reference points on the image; and
cropping another portion of the image corresponding to another ROI that is identified based on the one or more positions of the one or more reference points on the image.

12. The non-transitory computer readable storage medium of claim 11 ,
wherein the selecting of the one or more reference points comprises obtaining three-dimensional world coordinates of the one or more reference points based on a terrain map and the location of the vehicle,
wherein the one or more positions of the one or more reference points on the image are determined by projecting the three-dimensional world coordinates of the one or more reference points to the image by using a camera pose information associated with the image, and
wherein the camera pose information characterizes optical properties, orientation, or location of the camera.

13. The non-transitory computer readable storage medium of claim 11 ,
wherein a position of a single reference point on the image is a center point of the another ROI.

14. The non-transitory computer readable storage medium of claim 11 , wherein the one or more pre-determined distances include 500 meters, 750 meters, or 1000 meters.

15. An apparatus for image processing on an autonomous vehicle comprising a processor, configured to:
obtain an image from a camera located on a vehicle while the vehicle is being driven, wherein the image is associated with a time that describes when the image was obtained;
determine a location of the vehicle in a spatial region where the vehicle is being driven, wherein the location is associated with the time when the image was obtained;
determine presence of a plurality of region of interest (ROI) relative to the location of the vehicle;
determine, in the image, locations of one or more ROI from the plurality of ROI;
determine, after the determine the locations of the one or more ROI in the image, that the image includes, from the one or more ROI, a second ROI that is same as that selected or cropped in a second image that is obtained at a second time prior to the time when the image is obtained; and
crop a portion of the image associated with a first ROI from the one or more ROI, wherein the second ROI is excluded from the one or more ROI prior to the cropping the portion of the image associated with the first ROI.

16. The apparatus of claim 15 , wherein the processor is further configured to:
determine that the image includes, from the one or more ROI, a second ROI that is same as that selected or cropped in a second image that is obtained at a second time prior to the time when the image is obtained; and
determine that the second ROI is excluded from the one or more ROI prior to the cropping the portion of the image associated with the first ROI.

17. The apparatus of claim 16 , wherein the second image is obtained by the camera located on the vehicle.

18. The apparatus of claim 15 , wherein the image is from a first region towards which the vehicle is being driven, or wherein the image is from a second region to a side of the vehicle, or wherein the image is from a third region aw

Assignees

• Tusimple Inc

Inventors

• Ge Lingting
• Liu Siyuan
• Qian Chenzhe
• Wang Yijie
• Huang Zehua
• Hou Xiaodi

Classifications

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06V10/25 Primary

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• H04N23/64 Primary

  Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image · CPC title

• H04N23/951

  by using two or more images to influence resolution, frame rate or aspect ratio · CPC title