Using divergence to conduct log-based simulations

Patent metadata

Field	Value
Publication number	US-10713148-B2
Application number	US-201816056865-A
Country	US
Kind code	B2
Filing date	Aug 7, 2018
Priority date	Aug 7, 2018
Publication date	Jul 14, 2020
Grant date	Jul 14, 2020

Abstract

The disclosure relate to testing software for operating an autonomous vehicle. For instance, a first simulation may be run using log data and the software to control a first simulated vehicle. During this, one or more characteristics of the simulated vehicle may be compared with one or more characteristics of a vehicle from the log data. The comparison may be used to determine a divergence point for starting a timer. In addition, a second simulation may be run using the log data and the software to control a second simulated vehicle. The divergence point may be used to determine a handover time to allow the software to take control of the second simulated vehicle. Whether the software is able to continue through the first simulation before the timer expires without a particular type of event occurring and/or the second simulation without the particular type of event occurring is determined.

First claim

The invention claimed is:

1. A method of testing software for operating a vehicle in an autonomous driving mode, the method comprising:
running a first simulation using log data collected by a vehicle operating in an autonomous driving mode, wherein the first simulation is run using the software to control a first simulated vehicle;
comparing one or more characteristics of the first simulated vehicle with one or more characteristics of the vehicle from the log data in order to determine a divergence point;
using the divergence point to determine a handover time for a second simulation, the handover time corresponding to a time in the second simulation at which the software is allowed to take control of a second simulated vehicle of the second simulation;
running the second simulation using the log data, wherein the second simulation is run using the handover time and the software to control the second simulated vehicle; and
determining whether the software is able to complete the second simulation without a particular type of event occurring in the second simulation.

2. The method of claim 1 , wherein the particular type of event is a collision, and the method further comprises, when the collision is determined to have occurred, flagging the second simulation for further review.

3. The method of claim 1 , wherein determining the divergence point includes comparing a planned trajectory of the first simulated vehicle with a planned trajectory of the vehicle from the log data.

4. The method of claim 3 , wherein determining the divergence point includes determining when one or more of a location, speed or change in speed of the planned trajectory of the first simulated vehicle and the planned trajectory of the vehicle from the log data diverge more than some threshold amount over some period of time.

5. The method of claim 1 , wherein determining the divergence point includes determining when a location of the first simulated vehicle and a location of the vehicle from the log data diverge more than a threshold amount.

6. The method of claim 1 , wherein determining the divergence point includes determining when a location of the first simulated vehicle and a location of the vehicle from the log data diverge more than a threshold amount in a lateral direction relative to a direction of traffic in a lane in which the simulated vehicle is traveling in the first simulation.

7. The method of claim 1 , wherein determining the divergence point includes determining when a location of the first simulated vehicle and a location of the vehicle from the log data diverge more than a threshold amount in a longitudinal direction relative to a direction of traffic in a lane in which the simulated vehicle is traveling in the first simulation.

8. The method of claim 7 , wherein determining the divergence point includes determining when the location of the first simulated vehicle and the location of the vehicle from the log data diverge more than a threshold amount in a lateral direction relative to the direction of traffic in the lane.

9. The method of claim 1 , further comprising:
at the divergence point in the first simulation, starting a timer that expires during the first simulation; and
flagging the first simulation for review when the particular type of event occurs before the timer expires.

10. The method of claim 9 , further comprising, not flagging the first simulation for review if the particular type of event only occurs during the first simulation after the timer expires.

11. The method of claim 9 , wherein the particular type of event is a collision between the first simulated vehicle and an object in the first simulation.

12. The method of claim 9 , wherein the particular type of event includes the simulated vehicle exhibiting a particular type of maneuvering behavior.

13. A method of testing software for operating a vehicle in an autonomous driving mode, the method comprising:
running a simulation using log data collected by a vehicle operating in an autonomous driving mode, wherein the simulation is run using the software to control a simulated vehicle;
comparing one or more characteristics of the simulated vehicle with one or more characteristics of the vehicle from the log data in order to determine a divergence point;
at the divergence point in the simulation, starting a timer that expires during the simulation; and
determining whether the software is able to continue through the simulation without a particular event occurring in the simulation prior to the timer expiring.

14. The method of claim 13 , wherein the particular event is a collision between the simulated vehicle and an object in the simulation.

15. The method of claim 13 , further comprising, flagging the simulation for review if the particular event occurs before the timer expires.

16. The method of claim 13 , further comprising, not flagging the simulation for review if the particular event only occurs during the simulation after the timer expires.

17. The method of claim 15 , wherein the particular event is a collision between the simulated vehicle and an object in the simulation.

18. The method of claim 13 , wherein the particular event includes the simulated vehicle exhibiting a particular type of maneuvering behavior.

19. The method of claim 13 , wherein determining the divergence point includes determining when a location of the simulated vehicle and a location of the vehicle from the log data diverge more than a threshold amount.

20. The method of claim 13 , wherein determining the divergence point includes determining when a location of the simulated vehicle and a location of the vehicle from the log data diverge more than a threshold amount in at least one of a lateral direction or a longitudinal direction relative to a direction of traffic in a lane in which the simulated vehicle is traveling in the simulation.

Assignees

• Waymo Llc

Inventors

• Nygaard Carl
• Xu Yiwen
• Stout James

Classifications

• G06F11/3698 Primary

  Environments for analysis, debugging or testing of software · CPC title

• G06F11/3476

  Data logging (G06F11/14, G06F11/2205 take precedence) · CPC title

• G06F11/3692 Primary

  for test results analysis · CPC title

• G06F11/3696

  Methods or tools to render software testable · CPC title

• G06F11/3684

  for test design, e.g. generating new test cases · CPC title