Remote operation of vehicles in close-quarter environments

Patent metadata

Field	Value
Publication number	US-10908604-B2
Application number	US-201816221124-A
Country	US
Kind code	B2
Filing date	Dec 14, 2018
Priority date	Dec 14, 2018
Publication date	Feb 2, 2021
Grant date	Feb 2, 2021

Abstract

Techniques for remote operation of vehicles are presented. Remote operation can involve maneuvering, by a vehicle computer system, a vehicle based on driving instructions from a remote computer system. The vehicle computer system collects sensor data from a plurality of sensors onboard the vehicle. Based on the sensor data, data associated with a visual representation of the surrounding environment is generated and sent to the remote computer system. The generating of the data can involve fusing sensor data to identify one or more features of a surrounding environment. The generated data may be indicative of the identified features. The visual representation is output on a display device of the remote computer system. The driving instructions can be based on human input supplied in response to the visual representation. In certain embodiments, the vehicle is maneuvered toward a location within proximity of a parking space before performing automated parking.

First claim

What is claimed is:

1. A method comprising:
collecting, by a computer system of a vehicle equipped with a plurality of sensors, sensor data from the plurality of sensors;
generating, by the computer system based on the sensor data, data associated with a visual video representation of a surrounding environment, wherein the generating of the data associated with the video representation comprises fusing sensor data from a first sensor of the plurality of sensors with sensor data from at least a second sensor of the plurality of sensors to identify one or more features of the surrounding environment, and wherein the data associated with the video representation is indicative of the one or more features;
sending, by the computer system, the data associated with the video representation to a remote computer system through a wireless communication link;
monitoring a transmission speed of the wireless communication link;
switching from sending the data associated with the video representation to sending data associated with a three-dimensional reconstruction of the surrounding environment in response to determining that the transmission speed has dropped below a threshold, wherein the three-dimensional reconstruction is generated based on fused sensor data;
receiving, by the computer system, driving instructions from the remote computer system, the driving instructions based on human input supplied in response to the video representation or the three-dimensional reconstruction after the video representation or the three-dimensional reconstruction has been displayed at the remote computer system based on data sent through the wireless communication link; and
maneuvering, by the computer system, the vehicle toward a location within the surrounding environment according to the driving instructions.

2. The method of claim 1 , further comprising:
receiving, by the computer system, a parking instruction from the remote computer system after reaching the location, wherein the location is within proximity of a parking space; and
performing, by the computer system, automated parking in response to the parking instruction, the automated parking causing the vehicle to park into the parking space.

3. The method of claim 1 , wherein the one or more features include an object in the surrounding environment, and wherein the data associated with the video representation or the data associated with the three-dimensional reconstruction is indicative of at least one of the following attributes of the object:
a distance of the object from the vehicle;
a location of the object;
a trajectory of the object;
a speed of the object;
a shape of the object; or
a size of the object.

4. The method of claim 3 , wherein the video representation or the three-dimensional reconstruction includes one or more graphical elements corresponding to the at least one of the attributes of the object.

5. The method of claim 4 , wherein the one or more graphical elements include at least one of an outline around the object, a silhouette of the object, or a graphical indication of the distance of the object from the vehicle.

6. The method of claim 1 , wherein the video representation or the three-dimensional reconstruction includes a three-dimensional model of the vehicle such that a boundary of the vehicle in spatial relation to an object in the surrounding environment is visible when the video representation or the three-dimensional reconstruction is displayed.

7. The method of claim 1 , further comprising:
in response to detecting an interruption in the wireless communication link, performing, by the computer system, automatic speed reduction until the vehicle comes to a stop or until the wireless communication link becomes available again.

8. The method of claim 1 , wherein the one or more features include an object in the surrounding environment, the method further comprising:
performing, by the computer system, automated emergency braking in response to determining that a driving instruction received from the remote computer system would cause the vehicle to collide with the object.

9. The method of claim 1 , wherein the first sensor is a camera, and wherein each sensor of the at least a second sensor is one of the following:
an additional camera;
a LIDAR sensor;
a radar sensor; or
an ultrasonic sensor.

10. A computer system comprising:
one or more processors; and
one or more computer-readable storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to:
collect sensor data from a plurality of sensors located in a vehicle;
generate data associated with a video representation of a surrounding environment, wherein the generating of the data associated with the video representation comprises fusing sensor data from a first sensor of the plurality of sensors with sensor data from at least a second sensor of the plurality of sensors to identify one or more features of the surrounding environment, and wherein the data associated with the video representation is indicative of the one or more features;
send the data associated with the video representation to a remote computer system through a wireless communication link;
monitor a transmission speed of the wireless communication link;
switch from sending the data associated with the video representation to sending data associated with a three-dimensional reconstruction of the surrounding environment in response to determining that the transmission speed has dropped below a threshold, wherein the three-dimensional reconstruction is generated based on fused sensor data;
receive driving instructions from the remote computer system, the driving instructions based on human input supplied in response to the video representation or the three-dimensional reconstruction after the video representation or the three-dimensional reconstruction has been displayed at the remote computer system based on data sent through the wireless communication link; and
maneuver the vehicle toward a location within the surrounding environment according to the driving instructions.

11. The computer system of claim 10 , wherein the instructions further cause the one or more processors to:
receive a parking instruction from the remote computer system after reaching the location, wherein the location is within proximity of a parking space; and
perform automated parking in response to the parking instruction, the automated parking causing the vehicle to park into the parking space.

12. The computer system of claim 10 , wherein the one or more features include an object in the surrounding environment, and wherein the data associated with the video representation or the data associated with the three-dimensional reconstruction is indicative of at least one of the following attributes of the object:
a distance of the object from the vehicle;
a location of the object;
a trajectory of the object;
a speed of the object;
a shape of the object; or
a size of the object.

13. The computer system of claim 12 , wherein the video representation or the three-dimensional reconstruction includes one or more graphical elements corresponding to the at least one of the attributes of the object.

14. The computer system of claim 13 , wherein the one or more graphical elements include at least one of an outline around the object, a silhouette of the object, or a graphical indication of the distance from the object to the vehicle.

15. The computer system of claim 10 , wherein the video representation or the three-dimensional reconstruction includes a three-dimens

Assignees

• Valeo Schalter & Sensoren Gmbh

Inventors

• Rastoll Cyril
• Rezaei Shahram
• Joos Malte

Classifications

• G06V20/586

  of parking space · CPC title

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• B60R2300/303

  using joined images, e.g. multiple camera images · CPC title

• G01S17/931

  of land vehicles · CPC title

• B60R2300/406

  using wireless transmission · CPC title