Systems and methods for depth synthesis with transformer architectures

Patent metadata

Field	Value
Publication number	US-12430840-B2
Application number	US-202318156958-A
Country	US
Kind code	B2
Filing date	Jan 19, 2023
Priority date	Jan 19, 2023
Publication date	Sep 30, 2025
Grant date	Sep 30, 2025

Abstract

Systems and methods for enhanced computer vision capabilities, particularly including depth synthesis, which may be applicable to autonomous vehicle operation are described. A vehicle may be equipped with a geometric scene representation (GSR) architecture for synthesizing depth views at arbitrary viewpoints. The GSR architecture synthesizes depth views enable advanced functions, including depth interpolation and depth extrapolation. The GSR architecture implements functions (i.e., depth interpolation, depth extrapolation) that are useful for various computer vision applications for autonomous vehicles, such as predicting depth maps from unseen locations. For example, a vehicle includes a processor device synthesizing depth views at multiple viewpoints, where the multiple viewpoints are from image data of a surrounding environment for the vehicle. Further, the vehicle can have a controller device that receives depth views from the processor device and performs autonomous operations in response to analysis of the depth views.

First claim

What is claimed is:

1. A vehicle, comprising:
one or more processors; and
memory storing machine-executable instructions in non-transitory memory, which when executed by the one or more processors, cause the vehicle to:
synthesize depth views at multiple viewpoints, wherein synthesizing the depth views at the multiple viewpoints comprises:
encoding images of an environment surrounding the vehicle into image embeddings, wherein the images are obtained from the multiple viewpoints,
encoding intrinsic parameters and relatives poses of one or more cameras into camera embeddings, wherein the one or more cameras captured the images,
projecting the image embeddings and the camera embeddings onto a latent representation for the environment using cross-attention layers of a neural network,
conditioning the latent representation using self-attention layers of the neural network,
generating second camera encodings for arbitrary cameras at arbitrary relative poses, and
querying the conditioned latent representation using the second camera embeddings to synthesize the depth views; and

perform one or more autonomous operations in response to analysis of the synthesized depth views.

2. The vehicle of claim 1 , wherein the one or more processors comprise a geometric scene representation (GSR) component.

3. The vehicle of claim 2 , wherein synthesizing depth views comprises depth estimations, depth interpolations, and depth extrapolations.

4. The vehicle of claim 3 , wherein the depth extrapolations comprise completed unseen portions of a scene including the surrounding environment for the vehicle.

5. The vehicle of claim 4 , wherein the depth extrapolations comprise generated dense depth maps from the multiple viewpoints.

6. The vehicle of claim 5 , wherein performing the one or more autonomous operations is in response to the dense depth maps.

7. The vehicle of claim 1 , wherein the one or more processors comprise a computer vision component performing one or more computer visual capabilities for the one or more autonomous operations.

8. The vehicle of claim 7 , wherein the one or more computer visual capabilities comprise object detection.

9. The vehicle of claim 1 , wherein the vehicle comprises an autonomous vehicle.

Assignees

• Toyota Res Inst Inc
• Toyota Tech Institute At Chicago
• Toyota Motor Co Ltd

Inventors

• Guizilini Vitor
• Vasiljevic Igor
• Gaidon Adrien D
• Shakhnarovich Greg
• Walter Matthew
• Fang Jiading
• Ambrus Rares A

Classifications

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title

• G06T9/00

  Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00) · CPC title

• H04N13/282

  for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems · CPC title

• H04N13/161

  Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) · CPC title

• H04N13/128

  Adjusting depth or disparity · CPC title