Geometric 3D augmentations for transformer architectures

Patent metadata

Field	Value
Publication number	US-12488483-B2
Application number	US-202318110421-A
Country	US
Kind code	B2
Filing date	Feb 16, 2023
Priority date	Jul 25, 2022
Publication date	Dec 2, 2025
Grant date	Dec 2, 2025

Abstract

A method of generating additional supervision data to improve learning of a geometrically-consistent latent scene representation with a geometric scene representation architecture is provided. The method includes receiving, with a computing device, a latent scene representation encoding a pointcloud from images of a scene captured by a plurality of cameras each with known intrinsics and poses, generating a virtual camera having a viewpoint different from viewpoints of the plurality of cameras, projecting information from the pointcloud onto the viewpoint of the virtual camera, and decoding the latent scene representation based on the virtual camera thereby generating an RGB image and depth map corresponding to the viewpoint of the virtual camera for implementation as additional supervision data.

First claim

What is claimed is:

1 . A method of generating additional supervision data to improve learning of a geometrically-consistent latent scene representation with a geometric scene representation architecture, the method comprising:
receiving, with a computing device, a latent scene representation encoding a pointcloud from images of a scene captured by a plurality of cameras each with known intrinsics and poses; selecting one of the plurality of cameras; translating a pose of the selected camera; adjusting a viewing angle of the translated selected camera toward a center of the pointcloud; generating a virtual camera having a viewpoint different from viewpoints of the plurality of cameras; projecting information from the pointcloud onto the viewpoint of the virtual camera; and decoding the latent scene representation based on the virtual camera thereby generating an RGB image and depth map corresponding to the viewpoint of the virtual camera for implementation as additional supervision data.

2 . The method of claim 1 , wherein translating the pose of the selected camera comprises adding translation noise to the pose of the selected camera.

3 . The method of claim 1 , wherein the virtual camera is generated by:
selecting one of the plurality of cameras as a canonical camera, applying a rotation matrix to the canonical camera, and propagating a rotation and a translation offset of the canonical camera resulting from the rotation matrix to other ones of the plurality of cameras.

4 . The method of claim 1 , further comprising:
implementing the geometric scene representation architecture with the computing device; inputting the images of the scene captured by the plurality of cameras into the geometric scene representation architecture, wherein each camera of the plurality of cameras includes known embeddings; and encoding the images of the scene captured by the plurality of cameras, with the geometric scene representation architecture, into the latent scene representation.

5 . The method of claim 1 , further comprising training the geometric scene representation architecture by inputting the RGB image and the depth map corresponding to the viewpoint of the virtual camera.

6 . The method of claim 1 , further comprising:
querying the latent scene representation with a camera embedding; and decoding the latent scene representation based on the camera embedding, thereby generating an estimated depth map and an estimated RGB image based on the camera embedding.

7 . A system for generating additional supervision data to improve learning of a geometrically-consistent latent scene representation with a geometric scene representation architecture, the system comprising:
one or more processors; and a non-transitory, computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to:
receive a latent scene representation encoding a pointcloud from images of a scene captured by a plurality of cameras each with known intrinsics and poses;
select one of the plurality of cameras;
translate a pose of the selected camera;
adjust a viewing angle of the translated selected camera toward a center of the pointcloud;
generate a virtual camera having a viewpoint different from viewpoints of the plurality of cameras;
project information from the pointcloud onto the viewpoint of the virtual camera; and
decode the latent scene representation based on the virtual camera thereby generating an RGB image and depth map corresponding to the viewpoint of the virtual camera for implementation as additional supervision data.

8 . The system of claim 7 , wherein translating the pose of the selected camera comprises adding translation noise to the pose of the selected camera.

9 . The system of claim 7 , wherein the virtual camera is generated by:
selecting one of the plurality of cameras as a canonical camera, applying a rotation matrix to the canonical camera, and propagating a rotation and a translation offset of the canonical camera resulting from the rotation matrix to other ones of the plurality of cameras.

10 . The system of claim 7 , wherein the instructions further cause the one or more processors to:
implement the geometric scene representation architecture with the system; input the images of the scene captured by the plurality of cameras into the geometric scene representation architecture, wherein each camera of the plurality of cameras includes known embeddings; and encode the images of the scene captured by the plurality of cameras, with the geometric scene representation architecture, into the latent scene representation.

11 . The system of claim 7 , wherein the instructions further cause the one or more processors to train the geometric scene representation architecture by inputting the RGB image and the depth map corresponding to the viewpoint of the virtual camera.

12 . The system of claim 7 , wherein the instructions further cause the one or more processors to:
query the latent scene representation with a camera embedding; and decode the latent scene representation based on the camera embedding, thereby generating an estimated depth map and an estimated RGB image based on the camera embedding.

13 . A computing program product for generating additional supervision data to improve learning of a geometrically-consistent latent scene representation with a geometric scene representation architecture, the computing program product comprising machine-readable instructions stored on a non-transitory computer readable memory, which when executed by a computing device, causes the computing device to carry out steps comprising:
receiving, with the computing device, a latent scene representation encoding a pointcloud from images of a scene captured by a plurality of cameras each with known intrinsics and poses; selecting one of the plurality of cameras; translating a pose of the selected camera; adjusting a viewing angle of the translated selected camera toward a center of the pointcloud; generating a virtual camera having a viewpoint different from viewpoints of the plurality of cameras; projecting information from the pointcloud onto the viewpoint of the virtual camera; and decoding the latent scene representation based on the virtual camera thereby generating an RGB image and depth map corresponding to the viewpoint of the virtual camera for implementation as additional supervision data.

14 . The computing program product of claim 13 , wherein translating the pose of the selected camera comprises adding translation noise to the pose of the selected camera.

15 . The computing program product of claim 13 , wherein the virtual camera is generated by:
selecting one of the plurality of cameras as a canonical camera, applying a rotation matrix to the canonical camera, and propagating a rotation and a translation offset of the canonical camera resulting from the rotation matrix to other ones of the plurality of cameras.

16 . The computing program product of claim 13 , the steps caused to be carried out by
the computing device further comprising: implementing the geometric scene representation architecture with the computing device; inputting the images of the scene captured by the plurality of cameras into the geometric scene representation architecture, wherein each camera of the plurality of cameras includes known embeddings; and encoding the images of the scene captured by the plurality of cameras, with the geometric scene representation architecture, into the latent sce

Assignees

• Toyota Res Inst Inc
• Toyota Motor Co Ltd
• Toyota Tech Institute At Chicago

Inventors

• Guizilini Vitor
• Vasiljevic Igor
• Gaidon Adrien D
• Fang Jiading
• Shakhnarovich Gregory
• Walter Matthew R
• Ambrus Rares A

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T7/85

  Stereo camera calibration · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title