Continuous tracing and metric collection system

Patent metadata

Field	Value
Publication number	US-11880293-B1
Application number	US-201916696597-A
Country	US
Kind code	B1
Filing date	Nov 26, 2019
Priority date	Nov 26, 2019
Publication date	Jan 23, 2024
Grant date	Jan 23, 2024

Abstract

Techniques for capturing and recording processor events and scheduler data in a production system on a per processing resource basis are discussed herein. In some examples, a process metric collection component may be associated with the scheduler and the processing resource such that the process metric collection component can capture real time data associated with the processes or threads both executed by the processing resource and waiting to execute on the processing resource. The captured data may be used by the system to monitor operations.

First claim

What is claimed is:

1. A vehicle comprising:
a first processing resource;
one or more communication connections;
one or more non-transitory computer readable media storing instructions, that when executed by one or more processors, cause the one or more processors to perform operations comprising:
collecting scheduling data associated with a process from a scheduling component associated with the first processing resource, wherein the process is associated with one or more of localization, prediction, or perception;
generating a first time stamp associated with the scheduling data based on a local clock signals;
collecting execution data associated with the process from the first processing resource during execution of the process;
generating a second time stamp associated with the execution data based on the local clock signals;
collecting completion data associated with completion of the process by the first processing resource;
generating a third time stamp associated with the completion data based on the local clock signals;
receiving substantially simultaneously a local reference clock signal and a global reference clock signal, the local reference clock signal associated with the first processing resource and the global reference clock signal associated with the vehicle;
determining a difference between the local reference clock signal and the global reference clock signal;
adjusting the first time stamp, the second time stamp, and the third time stamp based at least in part on the difference;
generating at least one message including at least one of the scheduling data, the execution data, or the completion data; and
causing the one or more communication connections to send the message to a remote system via a network.

2. The vehicle of claim 1 , wherein the operations further comprise:
determining an execution threshold for the process based at least in part on speed of the vehicle and the scheduling data, execution data, and completion data associated with the process executing on the first processing resource.

3. The vehicle of claim 1 , wherein the operations further comprise:
determining a time associated with the execution of the process by the first processing resource exceeds an execution threshold; and
sending, in response to determining the time exceeded the execution threshold, an alert to an operational system of the vehicle.

4. A method comprising:
receiving, by a metric collection component, a first indication of an initialization of a process on a processing resource, wherein the process is associated with one or more of localization, prediction, or perception;
determining, by the metric collection component, first time data associated with the initialization on the processing resource;
receiving, by the metric collection component, a second indication of a process completion by the processing resource;
determining, by the metric collection component, second time data associated with the process completion by the processing resource;
receiving substantially simultaneously a local reference clock signal and a global reference clock signal, the local reference clock signal associated with the processing resource and the global reference clock signal associated with a system including the processing resource and at least one additional processing resource;
determining a difference between the local reference clock signal and the global reference clock signal;
adjusting the first time data and the second time data based at least in part on the difference;
generating a message comprising data associated with the process, the first time data, and the second time data; and
sending the message to a remote system.

5. The method of claim 4 , further comprising determining an execution threshold for the process based at least in part on a speed of a vehicle associated with the processing resource and the data associated with the process.

6. The method of claim 4 , wherein the data associated with the process comprises one or more of an identifier associated with the processing resource.

7. The method of claim 4 , wherein the data associated with the process comprises a total processing time, one or more process dependencies, one or more cache hits associated with the process, and one or more cache misses associated with the process.

8. The method of claim 4 , further comprising:
determining a time associated with an execution of the process by the processing resource exceeds an execution threshold, wherein the execution threshold is based on a speed of a vehicle including the processing resource; and
sending, in response to determining the time exceeded the execution threshold, an alert to an operational system of the vehicle.

9. The method of claim 8 , further comprising reducing, by the operational system, a speed of the vehicle in response to the time exceeding the execution threshold.

10. The method of claim 4 , wherein the processing resource is a first processing resource, the method further comprising:
receiving, by the metric collection component, a third indication of an initialization of a second process on a second processing resource, the second processing resource different than the first processing resource;
determining, by the metric collection component, third time data associated with the initialization of the second process on the second processing resource;
receiving, by the metric collection component, a fourth indication of a process completion of the second processing resource by the second processing resource;
determining, by the metric collection component, fourth time data associated with the process completion of the second process by the second processing resource;
receiving substantially simultaneously a second local reference clock signal and the global reference clock signal, the second local reference clock signal associated with the second processing resource;
determining a second difference between the second local reference clock signal and the global reference clock signal;
adjusting the third time data and the fourth time data based at least in part on the second difference;
generating a second message comprising data associated with the second process, the third time data, and the fourth time data; and
sending the second message to the remote system.

11. The method of claim 4 , further comprising:
receiving, by the metric collection component, a third indication associated with execution of the process from the processing resource;
determining, by the metric collection component, third time data associated with the execution of the process; and
wherein the message comprises the third time data.

12. The method of claim 4 , wherein the metric collection component is part of a kernel associated with the processing resource.

13. The method of claim 4 , further comprising sending an alert to a remote operator associated with an autonomous vehicle.

14. One or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform operations comprising:
identifying an individual process executing on the processor, the individual process associated with normal operation of a vehicle, wherein the individual process is associated with one or more of localization, prediction, or perception;
capturing data associated with the execution of the individual process;
receiving schedule data associated with the one or more processors;
determining that the execution of the individual process has exceeded an execution threshold;
generating an alert based at least in part on the data associated

Assignees

• Zoox Inc

Inventors

• Hendrix Austin
• King Andrew Lewis
• Flanagan Thomas Michael

Classifications

• G06F11/362 Primary

  Debugging of software · CPC title

• G06F11/3409

  for performance assessment · CPC title

• G06F11/302

  where the computing system component is a software system · CPC title

• G05D1/0011

  associated with a remote control arrangement · CPC title

• G05D1/0088

  characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00) · CPC title