Automatically associating road sign information with road segments using image data

Patent metadata

Field	Value
Publication number	US-10891501-B2
Application number	US-201916267697-A
Country	US
Kind code	B2
Filing date	Feb 5, 2019
Priority date	Jan 28, 2019
Publication date	Jan 12, 2021
Grant date	Jan 12, 2021

Abstract

A transport service system automatically associates road sign information with road segments based on image data including a representation of the road sign. The transport service system receives image data from one or more camera systems. Based on the image data, the transport service system determines a distance score and an orientation score for one or more candidate road segments. The transport service system additionally determines whether the image data is anomalous. Based on the determined distance and orientation scores, the transport service system generates a composite score and a confidence score for each candidate road segment. The confidence score is adjusted based on whether the image data is anomalous. Based on the generated scores, the transport service system identifies a most likely candidate road segment for association with the road sign.

First claim

What is claimed is:

1. A method for associating road signs with road segments in map data for a transport service system, the method comprising:
receiving image data including a representation of a road sign, a location, and a camera direction;
identifying road sign information from the representation of the road sign in the received image data;
identifying one or more candidate road segments, each candidate road segment potentially associated with the road sign information;
for each candidate road segment:
determining a distance score for the road sign and the candidate road segment;
determining an orientation score for the road sign and the candidate road segment; and
determining a composite score for the candidate road segment, the composite score based on the distance score and the orientation score for the road sign and the candidate road segment;

identifying a most likely candidate road segment for the road sign based on the determined composite scores; and
associating the identified most likely candidate road segment with the identified road sign information in map data for the transport service system.

2. The method of claim 1 , wherein identifying a most likely candidate road segment for the road sign based on the determined composite scores further comprises:
ranking the candidate road segments, the ranking based on the determined composite scores;
identifying a candidate road segment with the highest determined composite score; and
selecting the candidate road segment with the highest determined composite score as the most likely candidate road segment.

3. The method of claim 1 , further comprising:
determining for each candidate road segment, based at least in part on the determined composite score associated with the candidate road segment, a confidence score;
determining that image data associated with the road sign is anomalous; and
responsive to the determination that the image data is anomalous, adjusting the confidence score for the candidate road segment.

4. The method of claim 3 , wherein each image of the received set of image data is associated with a sighting triangle of a camera and determining that image data associated with the road sign is anomalous further comprises:
identifying that at least one image of the set of image data is associated with a sighting triangle that does not intersect with the sighting triangles associated with the other images in the set of image data; and
determining that the at least one image of the set of image data is anomalous.

5. The method of claim 3 , wherein each image of the received set of image data is associated with a sighting triangle of a camera and determining that image data associated with the road sign is anomalous further comprises:
identifying that at least one image of the set of image data is associated with a camera direction and sighting triangle indicating an inconsistent sign orientation to a sign orientation indicated in one or more other images of the set of image data; and
determining that the at least one image of the set of image data is anomalous.

6. The method of claim 3 , wherein adjusting the confidence score for the candidate road segment comprises reducing the score for the candidate road segment based on a set confidence factor.

7. The method of claim 1 , wherein identifying one or more candidate road segments, the one or more candidate road segments identified as potentially being described by the road sign information, further comprises:
accessing map data associated with an area based on the location of the image data; and
identifying one or more candidate road segments within a threshold distance from the location of the image data.

8. The method of claim 1 , wherein determining a distance score for the road sign and the candidate road segment further comprises:
determining an orthogonal distance from the road sign to the candidate road segment; and
determining a distance score for the road sign and the candidate road segment as a function of the orthogonal distance from the road sign to the candidate road segment.

9. The method of claim 1 , wherein determining an orientation score for the road sign and the candidate road segment further comprises:
determining an orientation for the road sign, the orientation based on the location associated with the image data and the camera direction associated with the image data; and
determining a relative angle between the orientation of the road sign and a vector of the candidate road segment; and
determining an orientation score for the road sign and the candidate road segment as a function of the relative angle between the orientation of the road sign and the vector of the candidate road segment.

10. The method of claim 1 , wherein the image data is received from a camera system mounted on a vehicle.

11. The method of claim 1 , determining a composite score for the candidate road segment comprises multiplicatively combining the distance score and the orientation score for the road sign and the candidate road segment.

12. A non-transitory computer-readable medium comprising computer program instructions executable by a processor to perform operations for associating road signs with road segments in map data for a transport service system, the operations comprising:
receiving image data including a representation of a road sign, the image data including information describing a location associated with the image data and a camera direction associated with the image data;
identifying road sign information from the representation of the road sign in the received image data;
identifying one or more candidate road segments, the one or more candidate road segments potentially being associated with the road sign information;
for each candidate road segment:
determining a distance score for the road sign and the candidate road segment;
determining an orientation score for the road sign and the candidate road segment; and
determining a composite score for the candidate road segment, the composite score based on the distance score and the orientation score for the road sign and the candidate road segment;
identifying a most likely candidate road segment for the road sign based on the determined composite scores; and
associating the identified most likely candidate road segment with the identified road sign information in map data for the transport service system.

13. The computer-readable storage medium of claim 12 , wherein identifying a most likely candidate road segment for the road sign based on the determined composite scores further comprises:
ranking the candidate road segments, the ranking based on the determined composite scores;
identifying a candidate road segment with the highest determined composite score; and
selecting the candidate road segment with the highest determined composite score as the most likely candidate road segment.

14. The computer-readable storage medium of claim 12 , further comprising:
determining for each candidate road segment, based at least in part on the determined composite score associated with the candidate road segment, a confidence score;
determining that image data associated with the road sign is anomalous; and
responsive to the determination that the image data is anomalous, adjusting the confidence score for the candidate road segment.

15. The computer-readable storage medium of claim 14 , wherein each image of the received set of image data is associated with a sighting triangle of a camera and determining that image data associated with the road sign is anomalous further comprises:

Assignees

• Uber Technologies Inc

Inventors

• He Lin
• Mohan Nishanth

Classifications

• G06V20/582 Primary

  of traffic signs · CPC title

• G06V2201/10

  Recognition assisted with metadata · CPC title

• G06K9/00818 Primary

  Physics · mapped topic