Systems and methods for compression of three-dimensional volumetric representations

Patent metadata

Field	Value
Publication number	US-12456229-B2
Application number	US-202017919460-A
Country	US
Kind code	B2
Filing date	Apr 17, 2020
Priority date	Apr 17, 2020
Publication date	Oct 28, 2025
Grant date	Oct 28, 2025

Abstract

Systems and methods are directed to encoding and/or decoding of the textures/geometry of a three-dimensional volumetric representation. An encoding computing system can obtain voxel blocks from a three-dimensional volumetric representation of an object. The encoding computing system can encode voxel blocks with a machine-learned voxel encoding model to obtain encoded voxel blocks. The encoding computing system can decode the encoded voxel blocks with a machine-learned voxel decoding model to obtain reconstructed voxel blocks. The encoding computing system can generate a reconstructed mesh representation of the object based at least in part on the one or more reconstructed voxel blocks. The encoding computing system can encode textures associated with the voxel blocks according to an encoding scheme and based at least in part on the reconstructed mesh representation of the object to obtain encoded textures.

First claim

What is claimed is:

1. A computer-implemented method to perform compression of three-dimensional volumetric representations, the method comprising:
obtaining, by an encoding computing system comprising one or more computing devices, one or more voxel blocks from a three-dimensional volumetric representation of an object, wherein the three-dimensional volumetric representation comprises a plurality of voxels and a respectively associated plurality of textures, wherein each of the plurality of voxels comprises a magnitude value and a sign value, and wherein each of the one or more voxel blocks comprising a subset of the plurality of voxels;
encoding, by the encoding computing system, the one or more voxel blocks with a machine-learned voxel encoding model to obtain one or more encoded voxel blocks;
decoding, by the encoding computing system, the one or more encoded voxel blocks with a first instance of a machine-learned voxel decoding model to obtain a first instance of one or more reconstructed voxel blocks;
generating, by the encoding computing system, a first instance of a reconstructed mesh representation of the object based at least in part on the first instance of the one or more reconstructed voxel blocks;
encoding, by the encoding computing system, at least a portion of the plurality of textures according to an encoding scheme, wherein each texture of the at least the portion of the plurality of textures is encoded based at least in part on a spatial position of each of the one or more reconstructed voxel blocks in three-dimensional space to obtain a plurality of encoded textures;
encoding, by the encoding computing system and using an entropy encoder, the one or more encoded voxel blocks to obtain one or more entropy encoded voxel blocks; and
transmitting, by the encoding computing system, the one or more entropy encoded voxel blocks, a voxel block index, and the plurality of encoded textures to a decoding computing system that is remotely located from the encoding computing system, wherein the voxel block index describes the spatial position of each of the one or more reconstructed voxel blocks in three-dimensional space.

2. The computer-implemented method of claim 1 , further comprising:
decoding, by a decoding computing system, the one or more encoded voxel blocks with a second instance of the machine-learned voxel decoding model to obtain a second instance of the one or more reconstructed voxel blocks;
decoding, by the decoding computing system according to the encoding scheme, the plurality of encoded textures to obtain a plurality of decoded textures; and
applying, by the decoding computing system, the plurality of decoded textures to the second instance of the one or more reconstructed voxel blocks to obtain a reconstructed three-dimensional volumetric representation of the object.

3. The computer-implemented method of claim 2 , wherein applying, by the decoding computing system, the plurality of decoded textures to the second instance of the one or more reconstructed voxel blocks to obtain the reconstructed three-dimensional volumetric representation of the object comprises:
generating, by the decoding computing system, a second instance of the reconstructed mesh representation of the object based at least in part on the second instance of the one or more reconstructed voxel blocks; and
applying, by the decoding computing system, the decoded plurality of textures to the second instance of the reconstructed mesh representation of the object to obtain the reconstructed three-dimensional volumetric representation.

4. The computer-implemented method of claim 3 , wherein the instances of the one or more reconstructed voxel blocks are obtained based at least in part on the voxel block index.

5. The computer-implemented method of claim 1 , wherein each voxel comprises a truncated signed distance field.

6. The computer-implemented method of claim 1 , wherein encoding, by the encoding computing system, the one or more voxel blocks with the machine-learned voxel encoding model to obtain the one or more encoded voxel blocks further comprises encoding, by the encoding computing system, the sign values of the one or more voxel blocks with the machine-learned voxel encoding model to obtain encoded sign values respectively associated with the one or more encoded voxel blocks, the encoded sign values based on a learned sign distribution, the learned sign distribution conditioned on the one or more encoded voxel blocks.

7. The computer-implemented method of claim 1 , wherein:
encoding, by the encoding computing system and using the entropy encoder, the one or more encoded voxel blocks to obtain the one or more entropy encoded voxel blocks further comprises encoding, by the encoding computing system and using the entropy encoder, the one or more encoded sign values to obtain entropy encoded sign values; and
transmitting, by the encoding computing system, the one or more entropy encoded voxel blocks, to the decoding computing system further comprises transmitting, by the encoding computing system, the entropy encoded sign values to the decoding computing system.

8. The computer-implemented method of claim 6 , wherein a decoding computing system decodes the one or more encoded voxel blocks and the one or more encoded sign values with a second instance of the machine-learned voxel decoding model to obtain the second instance of the one or more reconstructed voxel blocks.

9. The computer-implemented method of claim 7 , wherein the entropy encoded sign values comprise a lossless encoding of the sign values of the plurality of voxels.

10. The computer-implemented method of claim 6 , wherein:
the one or more encoded voxel blocks are encoded based at least in part on a learned voxel distribution;
at least one of the learned voxel distribution, the learned sign distribution, the machine-learned voxel encoding model, or the machine-learned voxel decoding model are trained on a loss function; and
the loss function evaluates a difference between the three-dimensional volumetric representation and the reconstructed three-dimensional volumetric representation.

11. The computer-implemented method of claim 1 , wherein encoding, by the encoding computing system, at least the portion of the plurality of textures according to the encoding scheme and based at least in part on the first instance of the reconstructed mesh representation of the object to obtain the plurality of encoded textures comprises:
extracting, by the encoding computing system, a plurality of polygons from the reconstructed mesh representation;
grouping, by the encoding computing system, the plurality of polygons into one or more polygon groups based at least in part on one or more polygon characteristics of each of the plurality of polygons;
generating, by the encoding computing system, a polygon chart comprising the one or more polygon groups, the polygon chart configured to maintain spatial coherence between each of the one or more polygon groups; and
mapping, by the encoding computing system, each of the one or more polygon charts to a texture atlas, wherein the spatial location of the one or more polygon charts in the texture atlas corresponds to the spatial location of each of the one or more reconstructed voxel blocks in three-dimensional space.

12. A computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store a first set of instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining one or more voxel blocks from a three-dimensional volumetric

Assignees

• Google Llc

Inventors

• Tang Danhang
• Singh Saurabh
• Keskin Cem
• Chou Phillip Andrew
• Haene Christian
• Dou Mingsong
• Fanello Sean Ryan Francesco
• Taylor Jonathan
• Tagliasacchi Andrea
• Davidson Philip Lindsley
• Zhang Yinda
• Guleryuz Onur Gonen
• Izadi Shahram
• Bouaziz Sofien

Classifications

• G06T9/002

  using neural networks · CPC title

• G06T17/30

  Polynomial surface description · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T9/001 Primary

  Model-based coding, e.g. wire frame · CPC title

• H04N19/597 Primary

  specially adapted for multi-view video sequence encoding · CPC title