Scene reliability

Patent metadata

Field	Value
Publication number	US-12459525-B2
Application number	US-202318344172-A
Country	US
Kind code	B2
Filing date	Jun 29, 2023
Priority date	Jun 29, 2023
Publication date	Nov 4, 2025
Grant date	Nov 4, 2025

Abstract

A computer implemented method is provided. The method, comprises: receiving from an autonomous vehicle sensor data, wherein the sensor data is indicative of an environment in which the vehicle is currently located or was previously located. The method further comprises receiving or determining additional data associated with at least one of: the sensor data, the vehicle, or the environment, wherein the additional data is different from the sensor data. The method further comprises displaying, on a display, an output comprising a representation of the sensor data. The method further comprises determining, based at least in part on the additional data, a reliability metric, the reliability metric being indicative of how reliable the sensor data is at representing the environment in which the vehicle is located at a current time. The method further comprises causing the output on the display to be based at least in part on the reliability metric.

First claim

What is claimed is:

1 . A system comprising:
one or more processors; and one or more non-transitory computer readable media having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving, at the system, from an autonomous vehicle remote from the system:
video data comprising video frame data, wherein the video frame data is indicative of an environment in which the vehicle is currently located or was previously located;

causing a display associated with a remote operator to output a representation of the video frame data;
receiving or determining, at the system, additional data associated with at least one of the vehicle or the environment;
determining a time difference associated with transmission of the video frame data from the autonomous vehicle to the system;
determining, based at least in part on the additional data and the time difference associated with the video frame data, a reliability metric, the reliability metric being indicative of how reliable the video frame data is at representing an environment in which the vehicle is located at a current time; and
causing the output on the display to be based at least in part on the reliability metric.

2 . The system of claim 1 , wherein causing the output on the display to be based at least in part on the reliability metric comprises causing the display to output a graphical item, wherein the presence of and/or characteristics of the graphical item are based on the reliability metric.

3 . The system of claim 2 , wherein the characteristics include at least one of: a form, size, color, position or opacity of the graphical item.

4 . The system of claim 1 , wherein the time difference is based on one of:
a time difference between a time at which the video frame data was recorded by the vehicle and the current time; a time difference between a time at which the video frame data was sent by the vehicle and the current time; a time difference between a time at which the video frame data was received by the system and the current time; or a time difference between a time at which the representation of the video frame data was output by the display and the current time; wherein a smaller time difference results in, or contributes to, a higher reliability metric and a greater time difference results in, or contributes to, a lower reliability metric; and wherein a higher reliability metric indicates that the video frame data is more reliable at representing the environment in which the vehicle is located at the current time than a lower reliability metric.

5 . The system of claim 1 , wherein data associated with the vehicle comprises at least one of:
a speed of the vehicle, wherein a lower speed results in, or contributes to, a higher reliability metric and a higher speed results in, or contributes to, a lower reliability metric, wherein a higher reliability metric indicates that the video frame data is more reliable at representing the environment in which the vehicle is located at the current time than a lower reliability metric; a distance travelled by the vehicle, wherein a lower distance results in, or contributes to, a higher reliability metric and a higher distance results in, or contributes to, a lower reliability metric; or a condition of one or more sensors of the vehicle, wherein a better condition results in, or contributes to, a higher reliability metric and a poorer condition results in, or contributes to, a lower reliability metric.

6 . The system of claim 1 , wherein data associated with the environment comprises at least one of:
a number of non-stationary objects in the environment, wherein a lower number of non-stationary objects results in, or contributes to, a higher reliability metric and a higher number of non-stationary objects results in, or contributes to, a lower reliability metric, wherein a higher reliability metric indicates that the video frame data is more reliable at representing the environment in which the vehicle is located at the current time than a lower reliability metric; a distance between the vehicle and one or more non-stationary objects in the environment, wherein a greater distance between the vehicle and the one or more non-stationary objects in the environment results in, or contributes to, a higher reliability metric and a smaller distance between the vehicle and the one or more non-stationary objects in the environment results in, or contributes to, a lower reliability metric; or a characteristic and/or state of one or more objects in the environment, wherein the characteristic and/or state of each object results in, or contributes to, a higher or lower reliability metric based on the object.

7 . A computer implemented method, comprising:
receiving, at a system, from an autonomous vehicle remote from the system:
sensor data, wherein the sensor data is indicative of an environment in which the vehicle is currently located or was previously located;
receiving or determining additional data associated with at least one of: the sensor data, the vehicle, or the environment; displaying, on a display, an output comprising a representation of the sensor data; determining, based at least in part on the additional data, a reliability metric, the reliability metric being indicative of how reliable the sensor data is at representing the environment in which the vehicle is located at a current time; and causing the output on the display to be based at least in part on the reliability metric.

8 . The method of claim 7 , wherein determining the reliability metric comprises determining the reliability metric from a reliability metric scale comprising a plurality of reliability metrics including at least a minimum reliability metric and a maximum reliability metric, the maximum reliability metric indicating that the sensor data is more reliable at representing the environment in which the vehicle is located at the current time than the minimum reliability metric.

9 . The method of claim 8 , wherein causing the output on the display to be based at least in part on the reliability metric comprises displaying, on the display, a graphical item, wherein the graphical item is based on the reliability metric, wherein a graphical item associated with the minimum reliability metric has a different form, size, color, position and/or opacity compared to a graphical item associated with the maximum reliability metric.

10 . The method of claim 7 , wherein the additional data comprises at least one of:
data associated with the vehicle, wherein the data associated with the vehicle comprises at least one of: a speed of the vehicle or a distance travelled by the vehicle or a location of the vehicle or a condition of one or more sensors of the vehicle; or data associated with the environment, wherein data associated with the environment comprises at least one of: data associated with a number of non-stationary objects in the environment or a distance between the vehicle and one or more non-stationary objects in the environment, or a characteristic and/or state of one or more objects in the environment.

11 . The method of claim 7 , wherein:
data associated with the sensor data comprises a timestamp, wherein the timestamp is based on one of: a time at which the sensor data was recorded by the vehicle, or a time at which the sensor data was sent by the vehicle, or a time at which the sensor data was received, or a time at which the representation of the sensor data was output by the display; the method further comprises determining, based on the timestamp and the current time, a time di

Assignees

• Zoox Inc

Inventors

• Chao Liang
• Choudry Muhammad Umar
• Cid Fernandez Gerardo
• Gogna Ravi
• Goldman Meredith James
• Orecchio Paul

Classifications

• G06T11/26

  Drawing of charts or graphs · CPC title

• B60W2756/10

  Involving external transmission of data to or from the vehicle · CPC title

• B60W2554/80

  Spatial relation or speed relative to objects · CPC title

• B60W2554/404

  Characteristics · CPC title

• B60W2050/146

  Display means · CPC title