System and method for occlusion detection in autonomous vehicle operation

Patent metadata

Field	Value
Publication number	US-12586385-B2
Application number	US-202318475647-A
Country	US
Kind code	B2
Filing date	Sep 27, 2023
Priority date	Feb 6, 2023
Publication date	Mar 24, 2026
Grant date	Mar 24, 2026

Abstract

The present disclosure provides methods and systems for operating an autonomous vehicle. In some embodiments, the system may obtain, by a camera associated with an autonomous vehicle, an image of an environment of the autonomous vehicle, the environment including a road on which the autonomous vehicle is operating and an occlusion on the road. The system may identify the occlusion in the image based on map information of the environment and at least one camera parameter of the camera for obtaining the image. The system may identify an object represented in the image, and determine a confidence score relating to the object. The confidence score may indicate a likelihood a representation of the object in the image is impacted by the occlusion. The system may determine an operation algorithm based on the confidence score; and cause the autonomous vehicle to operate based on the operation algorithm.

First claim

What is claimed is:

1 . A method of operating an autonomous vehicle, comprising:
obtaining, by a camera associated with an autonomous vehicle, a first image of an environment of the autonomous vehicle, the environment including a road on which the autonomous vehicle is operating and an occlusion on the road; identifying, by at least one processor, associated with the autonomous vehicle, the occlusion in the first image based on map information of the environment and at least one camera parameter of the camera for obtaining the first image; identifying, by the at least one processor, an object represented in the first image; determining, by the at least one processor, a confidence score relating to the object, wherein the confidence score indicates a likelihood a representation of the object in the first image is impacted by the occlusion; determining an operation algorithm based on the confidence score; and causing the autonomous vehicle to operate based on the operation algorithm; wherein:
the at least one camera parameter of the camera comprises at least one of a height of the camera or a tilt angle of the camera;
the first image comprises a reference line relating to the at least one camera parameter;
the map information comprises a contour of the road, and
identifying the occlusion in the first image based on the map information of the environment and at least one parameter of the camera comprises:
at each of a plurality of points along the reference line,
determining a distance between the point on the reference line and a corresponding location on the contour of the road; and

identifying a location of the occlusion based on the distances.

2 . The method of claim 1 , wherein the at least one camera parameter of the camera comprises at least one of an intrinsic camera parameter of the camera or an extrinsic camera parameter of the camera.

3 . The method of claim 1 , wherein the identifying the occlusion in the first image based on the map information of the environment and at least one parameter of the camera comprises:
marking the occlusion in the first image.

4 . The method of claim 3 , wherein:
the map information comprises a contour of the road, the occlusion comprises a change in the contour of the road in an occlusion direction, identifying the object represented in the first image comprises determining a bounding box that encloses a representation of the object in the first image; and determining the confidence score relating to the object comprises:
determining a distance between the bounding box and the occlusion marked in the first image along the occlusion direction; and
determining the confidence score based on the distance.

5 . The method of claim 4 , wherein determining the distance between the bounding box and the occlusion marked in the first image along the occlusion direction comprises:
identifying a reference point of the bounding box; and determining a pixel count of pixels along a line connecting the reference point of the bounding box and the occlusion marked in the first image along the occlusion direction.

6 . The method of claim 5 , wherein the reference point of the bounding box is a center of the bounding box.

7 . The method of claim 5 , wherein:
the occlusion direction is a vertical direction substantially perpendicular to a plane of the road.

8 . The method of claim 5 , wherein:
the occlusion direction is a lateral direction substantially in or parallel to a plane of the road.

9 . The method of claim 1 , wherein determining the operation algorithm based on the confidence score comprises:
in response to determining that the confidence score relating to the object is below a confidence score threshold, maintaining the operation algorithm currently in effect.

10 . The method of claim 1 , wherein determining the operation algorithm based on the confidence score comprises:
in response to determining that the confidence score exceeds a confidence score threshold, adjusting the operation algorithm currently in effect.

11 . The method of claim 10 , wherein:
the autonomous vehicle travels at a first speed when the autonomous vehicle operates according to the operation algorithm currently in effect and travels at a second speed when operating according to the adjusted operation algorithm, and the first speed is greater than the second speed.

12 . The method of claim 1 , wherein determining the operation algorithm based on the confidence score comprises:
selecting, based on the confidence score, an object detection model used for identifying the object in the first image.

13 . The method of claim 1 , further comprising obtaining, using the camera, a second image of the environment at a second time point, wherein:
the first image is obtained at a first time point different from the second time point, identifying the object in the first image is performed based on a first object detection model,
and
in response to determining that the confidence score exceeds a confidence score threshold, determining the operation algorithm comprises selecting a second object detection model for identifying the object in the second image, the second object detection model being different from the first object detection model.

14 . The method of claim 1 , further comprising:
obtaining, using the camera, a plurality of images of the environment at a plurality of time points; tracking the object based on the plurality of images; and determining the operation algorithm based further on the tracking of the object.

15 . An apparatus, comprising:
at least one processor; and at least one memory including computer program code which, when executed by the at least one processor, cause apparatus to perform operations comprising: obtaining, from a camera associated with an autonomous vehicle, a first image of an environment of the autonomous vehicle, the environment including a road on which the autonomous vehicle is operating and an occlusion on the road; identifying, by the at least one processor, the occlusion in the first image based on map information of the environment and at least one camera parameter of the camera for obtaining the first image; identifying, by the at least one processor, an object represented in the first image; determining, by the at least one processor, a confidence score relating to the object, wherein the confidence score indicates a likelihood a representation of the object in the first image is impacted by the occlusion; determining, by the at least one processor, an operation algorithm based on the confidence score; and causing the autonomous vehicle to operate based on the operation algorithm; wherein:
the at least one camera parameter of the camera comprises at least one of a height of the camera or a tilt angle of the camera;
the first image comprises a reference line relating to the at least one camera parameter;
the map information comprises a contour of the road, and
identifying the occlusion in the first image based on the map information of the environment and at least one parameter of the camera comprises:
at each of a plurality of points along the reference line, determining a distance between the point on the reference line and a corresponding location on the contour of the road; and
identifying a location of the occlusion based on the distances.

16 . The apparatus of claim 15 , wherein the camera is attached to a top surface of the autonomous vehicle.

17 . The apparatus of claim 15 , wherein at least one of the at leas

Assignees

• Tusimple Inc
• Createai Inc

Inventors

• Chen Zhe
• Zhao Yizhe
• Ge Lingting
• Wang Panqu

Classifications

• B60W2420/403

  Image sensing, e.g. optical camera · CPC title

• B60W60/001

  Planning or execution of driving tasks · CPC title

• B60W2552/15

  Road slope, i.e. the inclination of a road segment in the longitudinal direction · CPC title

• B60W2556/40

  High definition maps · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title