System and method for providing an interpretable and unified representation for trajectory prediction

Patent metadata

Field	Value
Publication number	US-11527073-B2
Application number	US-202016911661-A
Country	US
Kind code	B2
Filing date	Jun 25, 2020
Priority date	Nov 15, 2019
Publication date	Dec 13, 2022
Grant date	Dec 13, 2022

Abstract

A system and method for providing an interpretable and unified representation for trajectory prediction that includes receiving birds-eye image data associated with travel of at least one agent within a roadway environment. The system and method also include analyzing the birds-eye image data to determine a potential field associated with the roadway environment and analyzing the birds-eye image data to determine a potential field associated with a past trajectory of the at least one agent. The system and method further include predicting a future trajectory of the at least one agent based on analysis of the potential fields.

First claim

The invention claimed is:

1. A computer-implemented method for providing an interpretable and unified representation for trajectory prediction, comprising:
receiving birds-eye image data associated with travel of at least one agent within a roadway environment;
analyzing the birds-eye image data to determine a potential field associated with the roadway environment;
analyzing the birds-eye image data to determine a potential field associated with a past trajectory of the at least one agent; and
predicting a future trajectory of the at least one agent based on analysis of the potential fields, wherein a predicted direction and a predicted speed of motion of the at least one agent within the roadway environment are modeled to generate the future trajectory by recurrently moving past a location on a displacement field.

2. The computer-implemented method of claim 1 , wherein receiving the birds-eye image data includes communicating with a road-side equipment to receive the birds-eye image data, wherein the birds-eye image data is associated with at least one birds-eye image of the roadway environment captured by at least one camera of the road-side equipment.

3. The computer-implemented method of claim 2 , wherein analyzing the birds-eye image data to determine the potential field associated with the roadway environment includes assigning each pixel of the at least one birds-eye image with a potential field pixel value, wherein the potential field pixel value is a scalar value that represents a potential energy that represents locations at which motion of the at least one agent is generated towards based on a road structure of the roadway environment.

4. The computer-implemented method of claim 2 , wherein analyzing the birds-eye image data to determine the potential field associated with the past trajectory of the at least one agent includes analyzing past motion information that includes agent trajectory coordinates that are associated with fixed coordinates that pertain to positions of the at least one agent during at least one past time step, wherein the potential field associated with the past trajectory is analyzed to encode an inertial effect and a social effect on a travel of the at least one agent within the roadway environment.

5. The computer-implemented method of claim 2 , further including determining a motion field from the road structure of the roadway environment and a motion field from the past trajectory of the at least one agent based on the potential field associated with the roadway environment and the potential field associated with the past trajectory of the at least one agent.

6. The computer-implemented method of claim 5 , wherein the motion field from the road structure includes pixel values of the at least one birds-eye image that include vector values that pertain to a directional structure of roadways of the roadway environment, wherein the motion field from the past trajectory of the at least one agent includes pixel values that include vector values that pertain to a past direction of the at least one agent.

7. The computer-implemented method of claim 5 , further including merging the motion field from the road structure and the motion field from the past trajectory, wherein the motion fields are merged into a merged motion field that indicates future motion constraints of the at least one agent traveling within the roadway environment that are based on roadway structural constraints and trajectory constraints.

8. The computer-implemented method of claim 7 , further including outputting the predicted speed of motion of the at least one agent based on past trajectory lengths associated with the past trajectory of the at least one agent at a plurality of past time steps.

9. The computer-implemented method of claim 8 , wherein predicting the future trajectory of the at least one agent includes multiplying the merged motion field with the predicted speed of motion of the at least one agent to determine the displacement field, wherein the displacement field indicates trajectory prediction of the at least one agent based on the predicted direction as determined based on the past trajectory and a length of movement based on the predicted speed of motion within the roadway environment during at least one future time step.

10. A system for providing an interpretable and unified representation for trajectory prediction, comprising:
a memory storing instructions when executed by a processor cause the processor to:
receive birds-eye image data associated with travel of at least one agent within a roadway environment;
analyze the birds-eye image data to determine a potential field associated with the roadway environment;
analyze the birds-eye image data to determine a potential field associated with a past trajectory of the at least one agent; and
predict a future trajectory of the at least one agent based on analysis of the potential fields, wherein a predicted direction and a predicted speed of motion of the at least one agent within the roadway environment are modeled to generate the future trajectory by recurrently moving past a location on a displacement field.

11. The system of claim 10 , wherein receiving the birds-eye image data includes communicating with a road-side equipment to receive the birds-eye image data, wherein the birds-eye image data is associated with at least one birds-eye image of the roadway environment captured by at least one camera of the road-side equipment.

12. The system of claim 11 , wherein analyzing the birds-eye image data to determine the potential field associated with the roadway environment includes assigning each pixel of the at least one birds-eye image with a potential field pixel value, wherein the potential field pixel value is a scalar value that represents a potential energy that represents locations at which motion of the at least one agent is generated towards based on a road structure of the roadway environment.

13. The system of claim 11 , wherein analyzing the birds-eye image data to determine the potential field associated with the past trajectory of the at least one agent includes analyzing past motion information that includes agent trajectory coordinates that are associated with fixed coordinates that pertain to positions of the at least one agent during at least one past time step, wherein the potential field associated with the past trajectory is analyzed to encode an inertial effect and a social effect on a travel of the at least one agent within the roadway environment.

14. The system of claim 11 , further including determining a motion field from the road structure of the roadway environment and a motion field from the past trajectory of the at least one agent based on the potential field associated with the roadway environment and the potential field associated with the past trajectory of the at least one agent.

15. The system of claim 14 , wherein the motion field from the road structure includes pixel values of the at least one birds-eye image that include vector values that pertain to a directional structure of roadways of the roadway environment, wherein the motion field from the past trajectory of the at least one agent includes pixel values that include vector values that pertain to a past direction of the at least one agent.

16. The system of claim 14 , further including merging the motion field from the road structure and the motion field from the past trajectory, wherein the motion fields are merged into a merged motion field that indicates future motion constraints of the at least one agent traveling within the roadway envi

Assignees

• Honda Motor Co Ltd

Inventors

• Choi Chiho
• Su Shan

Classifications

• G06V20/17

  taken from planes or by drones · CPC title

• G06V10/806

  of extracted features · CPC title

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title

• G06V20/56 Primary

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06F18/253

  of extracted features · CPC title