Display augmentation

Patent metadata

Field	Value
Publication number	US-12566433-B2
Application number	US-202318452899-A
Country	US
Kind code	B2
Filing date	Aug 21, 2023
Priority date	Aug 21, 2023
Publication date	Mar 3, 2026
Grant date	Mar 3, 2026

Abstract

A method is provided, comprising: (i) receiving, at a system, from an autonomous vehicle: (a) sensor data captured by a sensor of the vehicle, and (b) output data generated based on environmental data captured by one or more sensors of the vehicle and used by a planning component to navigate in an environment, (ii) causing one or more displays to display at least one of: a representation of the sensor data, or a model of the environment based on the output data, (iii) determining a location of a feature within the sensor data or the model, (iv) causing the one or more displays to display an indication of the feature at a position corresponding to the location, (v) receiving, at the system, user input, and (vi) sending, by the system to the vehicle, data based on the user input to cause the vehicle to take an action.

First claim

What is claimed is:

1 . A system comprising:
one or more processors; and one or more non-transitory computer readable media having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations comprising: receiving, at the system, from a vehicle, the vehicle an autonomous vehicle:
video frame data captured by a video camera of the vehicle; and
perception output data generated by the vehicle, wherein the perception output data is generated based at least in part on environmental data captured by one or more sensors of the vehicle and is used by a planning component of the vehicle to navigate in an environment, wherein the video frame data and perception output data are indicative of the environment;
causing a first area of one or more displays to display a representation of the video frame data; causing a second area of the one or more displays to display a model of the environment based on the perception output data; receiving, at the system, user input, the user input being associated with a feature within one of the first area or the second area; determining a first position within the first area or the second area associated with the feature; determining that the feature is not present within the other of the first area or the second area; based at least in part on determining that the feature is not present within the other of the first area or the second area, causing the one or more displays to display in the other of the first area or the second area an indication of the feature at a second position; and sending, by the system to the vehicle, data based at least in part on the user input to cause the vehicle to take an action.

2 . The system of claim 1 , wherein:
the feature corresponds to a navigation path for the vehicle; the user input comprises user input corresponding to modifying or generating the navigation path for the vehicle; and the data sent by the system to the vehicle comprises data associated with the navigation path for the vehicle to follow the navigation path.

3 . The system of claim 1 , wherein:
the feature is visible within the displayed representation of the video frame data; causing the one or more displays to display in the other of the first area or the second area an indication of the feature at the second position comprises causing the one or more displays to display in the second area an indication of the feature at the second position; the operations further comprise generating additional perception output data for use by the planning component of the vehicle to navigate in the environment, wherein the additional perception output data is associated with the feature; and the data sent by the system to the vehicle comprises the additional perception output data, and the additional perception output data comprises:
data associated with the feature; and
a location of the feature within the environment based on the first position.

4 . The system of claim 1 , wherein:
the feature corresponds to an object within the environment; receiving, at the system, the user input, comprises receiving at least a first user input and a second user input, the first user input being associated with the first position within one of the first area or the second area of the one or more displays, and the second user input corresponding to a classification of the feature; and the data sent by the system to the vehicle comprises the classification.

5 . The system of claim 1 , wherein:
receiving, at the system, the user input, comprises receiving at least a first user input and a second user input, the first user input being associated with the first position within one of the first area or the second area of the one or more displays, and the second user input corresponding to a driving instruction for the vehicle; and the data sent by the system to the vehicle comprises the driving instruction that the vehicle uses to navigate in the environment.

6 . The system of claim 1 , wherein:
the video frame data and the model are associated with a coordinate system; determining, based at least in part on the first position, the second position of the feature within the other of the first area or the second area, comprises:
determining a location of the feature within the coordinate system, based on the first position; and
determining the second position based on the location.

7 . A method, comprising:
receiving, at a system, from a vehicle, the vehicle an autonomous vehicle:
sensor data captured by a sensor of the vehicle; and
output data, wherein the output data is generated based at least in part on environmental data captured by one or more sensors of the vehicle and used by a planning component of the vehicle to navigate in an environment, wherein the sensor data and the output data are indicative of the environment;
causing one or more displays to display at least one of: (i) a representation of the sensor data, or (ii) a model of the environment based on the output data; automatically determining a location of a feature within the sensor data or the model, the feature a first feature of a plurality of features in the environment; determining that at least the first feature of the plurality of features is not present in both the sensor data and the model; causing the one or more displays to display an indication of the feature at a position on the one or more displays corresponding to the location based at least in part on determining that at least the first feature of the plurality of features is not present in both the sensor data and the model; receiving, at the system, user input; and sending, by the system to the vehicle, data based on the user input to cause the vehicle to take an action.

8 . The method of claim 7 , wherein:
the feature corresponds to a priority agent; causing the one or more displays to display at least one of: (i) a representation of the sensor data, or (ii) the model and causing the one or more displays to display an indication of the feature comprises:
causing the one or more displays to display at least the representation of the sensor data; and
causing the one or more displays to display an indication of the priority agent within the representation of the sensor data; and
the data sent by the system to the vehicle comprises data associated with the priority agent.

9 . The method of claim 7 , wherein:
causing the one or more displays to display at least one of: (i) the representation of the sensor data, or (ii) the model, comprises:
causing the one or more displays to display both: (i) the representation of the sensor data, and (ii) the model; and
causing the one or more displays to display both: (i) the representation of the sensor data, and (ii) the model, comprises:
causing the one or more displays to display the model such that it has a perspective corresponding a perspective of the representation of the sensor data.

10 . The method of claim 7 , wherein:
causing the one or more displays to display at least one of: (i) the representation of the sensor data, or (ii) the model, comprises:
causing the one or more displays to display both: (i) the representation of the sensor data, and (ii) the model;
receiving, at the system, the user input, comprises receiving at least a first user input, the first user input being associated with the feature and a first position on the one or more displays within the displayed representation of the sensor data or the displayed model; receiving the first user input causes the location of the feature within the sensor data or the model to be determined based on the first posit

Assignees

• Zoox Inc

Inventors

• Choudry Muhammad Umar
• Goldman Meredith James

Classifications

• G05D2109/10

  Land vehicles · CPC title

• G05D2105/22

  of humans · CPC title

• G05D2107/13

  Spaces reserved for vehicle traffic, e.g. roads, regulated airspace or regulated waters · CPC title

• G05D1/2246

  displaying a map of the environment · CPC title

• G05D1/2249

  using augmented reality · CPC title