Road defect detection

Patent metadata

Field	Value
Publication number	US-12530909-B2
Application number	US-202318497254-A
Country	US
Kind code	B2
Filing date	Oct 30, 2023
Priority date	Oct 30, 2023
Publication date	Jan 20, 2026
Grant date	Jan 20, 2026

Abstract

A system for identifying road defects within a roadway ahead of a moving vehicle, includes at least one camera in communication with a system controller and adapted to capture a plurality of time sequential images of the roadway ahead of the moving vehicle, and at least one motion sensor in communication with the system controller and adapted to detect when the vehicle encounters a road defect, the system controller adapted to correlate the road defect with at least one of the plurality of time sequential images of the roadway, analyze, using a computer vision algorithm, the at least one of the plurality of time sequential images, identify, with the computer vision algorithm, the road defect within the at least one of the plurality of time sequential images, and label the identified road defect within the at least one of the plurality of time sequential images.

First claim

What is claimed is:

1 . A method of identifying road defects within a roadway ahead of a moving vehicle, comprising:
capturing, with at least one camera in communication with a system controller, a plurality of time sequential images of the roadway ahead of the moving vehicle; detecting, with at least one motion sensor positioned within the vehicle and in communication with the system controller, when the vehicle encounters a road defect; correlating, with the system controller, the road defect detected by the at least one motion sensor with at least one of the plurality of time sequential images of the roadway collected by the at least one camera; analyzing, with the system controller, using a computer vision algorithm, the at least one of the plurality of time sequential images that is correlated to the road defect detected by the at least one motion sensor; identifying, with the system controller and the computer vision algorithm, the road defect within the at least one of the plurality of time sequential images that is correlated to the road defect detected by the at least one motion sensor; labeling, with the system controller, the identified road defect within the at least one of the plurality of time sequential images that is correlated to the road defect detected by the at least one motion sensor; and determining, with the system controller, a lateral position of the road defect within the roadway, including:
receiving, with the system controller, from the at least one motion sensor, data related to a relative severity of the road defect at different lateral locations across the roadway, wherein the at least one motion sensor includes a plurality of motion sensors located at different lateral positions within the vehicle;
receiving, with the system controller, from a remote database in communication with the system controller, data related to a relative severity of the road defect at different lateral locations across the roadway collected previously by other vehicles;
predicting, with the system controller, using a global machine learning model in communication with the system controller and the remote database and trained with data related to a relative severity of the road defect at different lateral locations across the roadway collected previously by other vehicles stored within the remote database, a lateral severity distribution for the road defect; and
updating the global machine learning model with data received from the plurality of motion sensors located at different lateral positions within the vehicle.

2 . The method of claim 1 , wherein the identifying, with the system controller and the computer vision algorithm, the road defect within the at least one of the plurality of time sequential images that is correlated to the road defect detected by the at least one motion sensor further includes:
receiving, with the system controller, from a local database in communication with the system controller, data from images previously captured by the vehicle wherein the road defect has been labelled by the system controller; and predicting, with the system controller, using a machine learning model in communication with the system controller and the local database, the location of the road defect in the at least one of the plurality of time sequential images that is correlated to the road defect detected by the at least one motion sensor.

3 . The method of claim 2 , further including updating the machine learning model with data of the detected road defect from the at least one motion sensor and the at least one of the plurality of time sequential images that is correlated to the road defect detected by the at least one motion sensor.

4 . The method of claim 1 , wherein the identifying, with the system controller and the computer vision algorithm, the road defect within the at least one of the plurality of time sequential images that is correlated to the road defect detected by the at least one motion sensor further includes:
receiving, with the system controller, from a remote database in communication with the system controller, data from images previously captured by other vehicles wherein the road defect has been identified and labelled; and predicting, with the system controller, using a global machine learning model in communication with the system controller and the remote database and trained with data from other vehicles stored within the remote database, the location of the road defect in the at least one of the plurality of time sequential images that is correlated to the road defect detected by the at least one motion sensor.

5 . The method of claim 4 , further including updating the global machine learning model with data of the detected road defect from the at least one motion sensor and the at least one of the plurality of time sequential images that is correlated to the road defect detected by the at least one motion sensor.

6 . The method of claim 1 , wherein the identifying, with the system controller and the computer vision algorithm, the road defect within the at least one of the plurality of time sequential images that is correlated to the road defect detected by the at least one motion sensor further includes:
receiving, with the system controller, from a remote database in communication with the system controller, data from images previously captured by other vehicles with contextual features matching the vehicle wherein the road defect has been identified and labelled; and predicting, with the system controller, using a contextual global machine learning model in communication with the system controller and the remote database and trained with data from other vehicle stored within the remote database, the location of the road defect in the at least one of the plurality of time sequential images that is correlated to the road defect detected by the at least one motion sensor.

7 . The method of claim 1 , wherein the correlating, with the system controller, the road defect detected by the at least one motion sensor with at least one of the plurality of time sequential images of the roadway collected by the at least one camera further includes:
detecting, at time tn, with the at least one motion sensor, the road defect; and calculating, with a position algorithm in communication with the system controller: a pose of the at least one camera at time tn; a relative position of the road defect relative to the at least one motion sensor and the at least one camera at time tn; a pose of the at least one camera for the at least one of the plurality of time sequential images that is correlated to the road defect detected by the at least one motion sensor, at time tn−x; and a relative position of the road defect within the at least one of the plurality of time sequential images that is correlated to the road defect detected by the at least one motion sensor, at time tn−x.

8 . The method of claim 1 , wherein the detecting, with at least one motion sensor positioned within the vehicle and in communication with the system controller, when the vehicle encounters a road defect further includes detecting road roughness and impact with a road defect with at least one motion sensor adapted to collect data related to wheel rotational differences.

9 . The method of claim 1 , wherein the detecting, with at least one motion sensor positioned within the vehicle and in communication with the system controller, when the vehicle encounters a road defect further includes collecting data related to a road defect with at least one inertial sensor.

10 . The method of claim 1 , wherein the detecting, with at least one motion sensor positioned within the vehicle and in communica

Assignees

• Gm Global Tech Operations Llc

Inventors

• Ali Kamran
• Grimm Donald K
• Yu Bo

Classifications

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/30184

  Infrastructure · CPC title

• G06T2207/30256

  Lane; Road marking · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title