Learning-oriented transferable automatic driving method and system driven by world model

Patent metadata

Field	Value
Publication number	US-12509112-B2
Application number	US-202318513257-A
Country	US
Kind code	B2
Filing date	Nov 17, 2023
Priority date	May 25, 2023
Publication date	Dec 30, 2025
Grant date	Dec 30, 2025

Abstract

The present invention relates to a learning-oriented transferable automatic driving method and system driven by a world model. The method includes the following steps: constructing a simulation environment; collecting a large batch of data in the simulation environment, and collecting a small batch of data in the real environment; constructing a world model, and performing model training in the simulation environment by using the large batch of data, wherein the world model takes a camera image as an input to model an environment by using a generative world model; storing and transmitting historical moment information by hidden variables, and outputting an aerial view and a control instruction; and performing domain adaptive transferring training in the real environment on the basis of the small batch of data, and deploying the model in an autonomous vehicle in the real world to achieve virtuality-to-reality transferring general integrated automatic driving.

First claim

What is claimed is:

1 . A learning-oriented transferable automatic driving method driven by a world model, comprising the following steps:
constructing a simulation environment, and setting, in the simulation environment, same automatic driving vehicle configurations as the real environment; collecting a first amount of data in the simulation environment, and collecting a second amount of data in the real environment, wherein the first amount of data is greater in quantity than the second amount of data, wherein the first data and the second data comprise bird's eye views and camera images captured by cameras from the front, back, left, and right directions of an automatic driving vehicle; the bird's eye view is a predefined world thumbnail, in which traffic vehicles and pedestrians are all marked with rectangular frames, and red traffic lights are marked with lines in a stop line region; constructing a world model and training the world model in the simulation environment by using the first amount of data, wherein the world model takes camera images as input and models an environment by using a generative world model; storing and transmitting historical moment information by hidden variables, and outputting a bird's eye view and a control instruction, wherein the control instruction is used for controlling the autonomous vehicle; performing domain adaptive transferring training in the real environment on the basis of the second amount of data by using the trained world model; and deploying the model completing the adaptive transferring training in the autonomous vehicle in the real world to achieve a general and integrated virtuality-to-reality transferring for automatic driving.

2 . The learning-oriented transferable automatic driving method driven by a world model according to claim 1 , wherein the automatic driving vehicle configurations comprise vehicle parameters and sensor parameters.

3 . The learning-oriented transferable automatic driving method driven by a world model according to claim 1 , wherein the world model at historical and current moments is expressed as:

{

x
k

=

f
e

(

o
k

)

q
⁡
(

s
k

)

∼

N
⁡
(

μ
θ

(

h
k

,

a

k
-
1

,

x
k

)

,

σ
θ

(

h
k

,

a

k
-
1

,

x
k

)

)

p
⁡
(

z
k

)

∼

N
⁡
(

μ
φ

(

h
k

,

a

k
-
1

)

,

σ
φ

(

h
k

,

a

k
-
1

)

)

h

k
+
1

=

f
ϕ

Assignees

• Univ Tongji

Inventors

• Huang Yanjun
• Du Jiatong
• Yang Shuo
• Bai Yulong
• Li Xincheng
• Chen Hong

Classifications

• B60W2420/403

  Image sensing, e.g. optical camera · CPC title

• B60W2050/0083

  Setting, resetting, calibration · CPC title

• B60W2050/0028

  Mathematical models, e.g. for simulation · CPC title

• B60W50/0097

  Predicting future conditions · CPC title

• G06V10/82

  using neural networks · CPC title