Using simulations to identify differences between behaviors of manually-driven and autonomous vehicles

Patent metadata

Field	Value
Publication number	US-12565233-B2
Application number	US-202318504670-A
Country	US
Kind code	B2
Filing date	Nov 8, 2023
Priority date	Jun 26, 2020
Publication date	Mar 3, 2026
Grant date	Mar 3, 2026

Abstract

A simulation may be used to determine a difference between progress of a manually-driven vehicle and progress of a simulated autonomous vehicle. The method includes retrieving log data collected for the manually-driven vehicle driving along a route, generating a plurality of path segments for a portion of the route. The plurality of path segments corresponds to points in a lane that the manually-driven vehicle traveled through on the portion of the route. The method also includes running, using a software of the autonomous vehicle, a simulation of the autonomous vehicle driving along the plurality of path segments, extracting metrics from the log data and the simulation, and determining the difference between a first progress of the manually-driven vehicle and a second progress of the simulated autonomous vehicle based on the metrics.

First claim

The invention claimed is:

1 . A method comprising:
receiving, by one or more processors of one or more server computing devices, log data of a manually-driven vehicle driven along a portion of a route; identifying, by the one or more processors, a plurality of path segments corresponding to the portion of the route; based on the identifying, running, by the one or more processors, using software for autonomous driving, a simulation of a simulated autonomous vehicle driving along the identified plurality of path segments; collecting, by the one or more processors, simulation data for the simulated autonomous vehicle while driving along the identified plurality of path segments as the simulation is run using the software for autonomous driving; comparing, by the one or more processors, the log data with the collected simulation data; and adjusting, by the one or more processors, the software for autonomous driving based on the comparing.

2 . The method of claim 1 , wherein comparing the log data with the collected simulation data comprises comparing performance of the software for autonomous driving.

3 . The method of claim 2 , further comprising:
based on the comparing, determining, by the one or more processors, differences between behaviors of the simulated autonomous vehicle and the manually-driven vehicle.

4 . The method of claim 3 , wherein the differences between behaviors indicate how the software for autonomous driving handles intersections as compared to a human driver.

5 . The method of claim 3 , wherein the differences between behaviors indicate how the software for autonomous driving handles stop signs as compared to a human driver.

6 . The method of claim 3 , wherein the differences between behaviors indicate how the manually-driven vehicle performs when responding to and interacting with one or more pedestrians in comparison to the software for autonomous driving.

7 . The method of claim 6 , wherein the log data identifies one or more of a shape, a speed, a location, and an orientation of the one or more pedestrians.

8 . The method of claim 6 , wherein the software for autonomous driving is adjusted based on how the manually-driven vehicle performs when responding to and interacting with the one or more pedestrians.

9 . The method of claim 1 , wherein adjusting the software for autonomous driving comprises altering a parameter for a type of maneuver associated with the portion of the route.

10 . The method of claim 1 , wherein the log data comprises at least one of sensor data or event data.

11 . The method of claim 1 , wherein the log data comprises data identifying characteristics of perceived objects, the perceived objects including any of pedestrians or bicyclists.

12 . The method of claim 1 , further comprising:
extracting, by the one or more processors, one or more metrics from the log data of the manually-driven vehicle and one or more metrics from the collected simulation data of the simulated autonomous vehicle.

13 . The method of claim 12 , wherein the one or more metrics include at least one of a distance metric, a time metric, a speed metric, or a point in time metric of an object reaching various points along the route.

14 . A non-transitory, tangible computer-readable medium on which computer-readable instructions of a program are stored, the instructions, when executed by one or more processors of one or more server computing devices, cause the one or more processors to perform a method comprising:
receiving log data of a manually-driven vehicle driven along a portion of a route; identifying a plurality of path segments corresponding to the portion of the route; based on the identifying, running, using software for autonomous driving, a simulation of a simulated autonomous vehicle driving along the identified plurality of path segments; collecting simulation data for the simulated autonomous vehicle while driving along the identified plurality of path segments as the simulation is run using the software for autonomous driving; comparing the log data with the collected simulation data; and adjusting the software for autonomous driving based on the comparing.

15 . The non-transitory, tangible computer-readable medium of claim 14 , wherein comparing the log data with the collected simulation data comprises comparing performance of the software for autonomous driving.

16 . The non-transitory, tangible computer-readable medium of claim 15 , wherein the method further comprises:
based on the comparing, determining differences between behaviors of the simulated autonomous vehicle and the manually-driven vehicle.

17 . The non-transitory, tangible computer-readable medium of claim 16 , wherein the differences between behaviors indicate how the software for autonomous driving handles intersections or stop signs as compared to a human driver.

18 . The non-transitory, tangible computer-readable medium of claim 16 , wherein the differences in behavior indicate how the manually-driven vehicle performs when responding to and interacting with one or more pedestrians in comparison to the software for autonomous driving.

19 . The non-transitory, tangible computer-readable medium of claim 18 , wherein the log data identifies one or more of a shape, a speed, a location, and an orientation of the one or more pedestrians.

20 . The non-transitory, tangible computer-readable medium of claim 18 , wherein the software for autonomous driving is adjusted based on how the manually-driven vehicle performs.

21 . The method of claim 1 , wherein the software for autonomous driving is prevented from taking a different path between two end points of each of the plurality of path segments.

22 . The method of claim 1 , further comprising determining a difference between a first progress of the manually-driven vehicle along the plurality of path segments and a second progress of the simulated autonomous vehicle along the plurality of path segments.

Assignees

• Waymo Llc

Inventors

• Cohen Brian
• Dupre Guillaume
• Russell Jared
• Schoenfeld Eric
• Wanderman Jack Joseph

Classifications

• B60W60/0013

  specially adapted for occupant comfort · CPC title

• G01C22/00 Primary

  Measuring distance traversed on the ground by vehicles, persons, animals or other moving solid bodies, e.g. using odometers, using pedometers · CPC title

• G01C21/3453

  Special cost functions, i.e. other than distance or default speed limit of road segments · CPC title

• G01C21/3407

  specially adapted for specific applications · CPC title

• G05D1/0088

  characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00) · CPC title