World-space segmentation

Patent metadata

Field	Value
Publication number	US-12100158-B2
Application number	US-202318205353-A
Country	US
Kind code	B2
Filing date	Jun 2, 2023
Priority date	Nov 19, 2019
Publication date	Sep 24, 2024
Grant date	Sep 24, 2024

Abstract

Methods, systems, and devices for generating a persistent world-space ground (or floor) segmentation map (or “texture”) for use in augmented or virtual reality 3D experiences.

First claim

What is claimed is:

1. A computer-implemented method comprising:
determining a segmentation for a first image frame of a plurality of image frames;
iteratively repeating the determination of the segmentation for each image frame following the first image frame until determining a sufficiency condition is met;
based on determining that the sufficiency condition is met, ceasing execution of determination of segmentation of each frame following the first image frame and continuing to use a last determined segmentation in subsequent iterations of image frames in the plurality of image frames; and
generating user interface data for display using the segmentation.

2. The computer-implemented method of claim 1 , wherein generating user interface data for display comprises:
generating a real-time virtual effect in screen-space using a world-space map and the segmentation; and
displaying the real-time virtual effect in a current image.

3. The computer-implemented method of claim 2 , wherein the world-space map is established by applying a surface tracker to the plurality of image frames.

4. The computer-implemented method of claim 1 further comprising:
initializing a segmentation texture for storing segmentation information prior to determining the segmentation.

5. The computer-implemented method of claim 4 , wherein the segmentation information relates to a fixed area around at least one image capture device.

6. The computer-implemented method of claim 5 , wherein the segmentation texture is a two-dimensional data structure, each entry in the segmentation texture including at least one value indicating a likelihood that a corresponding pixel in an image frame fulfils a segmentation criterion.

7. The computer-implemented method of claim 6 , wherein the segmentation texture is a ground/floor segmentation texture and the segmentation criterion is a criterion for assigning a pixel to a ground/floor object class.

8. The computer-implemented method of claim 1 , wherein determining the segmentation comprises:
obtaining segmentation in screen-space from at least one image frame of the plurality of image frames;
projecting the screen-space segmentation into world-space; and
integrating the projected screen-space segmentation into a world-space map using a temporal filter.

9. The computer-implemented method of claim 8 , wherein segmentation is obtained in screen space by executing a semantic segmentation neural network with the at least one image frame of the plurality of image frames as input.

10. The computer-implemented method of claim 8 , wherein determining the segmentation further comprises:
storing the screen-space segmentation in a segmentation texture.

11. The computer-implemented method of claim 8 , wherein the screen-space segmentation is projected into world-space by storing the projected screen-space segmentation in a temporary world-space segmentation texture and then integrating the temporary world-space segmentation texture from this frame into the world-space map.

12. The computer-implemented method of claim 8 , wherein the temporal filter is an infinite impulse response (IIR) filter or a finite impulse response (FIR) filter.

13. The computer-implemented method of claim 1 , further comprising:
running at least one world-space post-processing filter.

14. The computer-implemented method of claim 1 , wherein the sufficiency condition is a requirement that a variance for a pixel is less than a variance threshold.

15. The computer-implemented method of claim 1 , wherein the plurality of image frames are obtained from a forward and rear camera of at least one image capture device.

16. A computing apparatus, the computing apparatus comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the computing apparatus to perform operations comprising:
determining a segmentation for a first image frame of a plurality of image frames;
iteratively repeating the determination of the segmentation for each image frame following the first image frame until determining a sufficiency condition is met;
based on determining that the sufficiency condition is met, ceasing execution of determination of segmentation of each frame following the first image frame and continuing to use a last determined segmentation in subsequent iterations of image frames in the plurality of image frames; and
generating user interface data for display using the segmentation.

17. The computing apparatus of claim 16 , wherein generating user interface data for display comprises:
generating a real-time virtual effect in screen-space using a world-space map and the segmentation; and
displaying the real-time virtual effect in a current image.

18. The computing apparatus of claim 16 , the operations further comprising:
initializing a segmentation texture for storing segmentation information prior to determining the segmentation, wherein the segmentation information relates to a fixed area around at least one image capture device.

19. The computing apparatus of claim 16 , the operations further comprising:
initializing a segmentation texture for storing segmentation information prior to determining the segmentation; and
wherein the segmentation texture is a two-dimensional data structure, each entry in the segmentation texture including at least one value indicating a likelihood that a corresponding pixel in an image frame fulfils a segmentation criterion, or
wherein the segmentation texture is a ground/floor segmentation texture and the segmentation criterion is a criterion for assigning a pixel to a ground/floor object class.

20. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising:
determining a segmentation for a first image frame of a plurality of image frames;
iteratively repeating the determination of the segmentation for each image frame following the first image frame until determining a sufficiency condition is met;
based on determining that the sufficiency condition is met, ceasing execution of determination of segmentation of each frame following the first image frame and continuing to use a last determined segmentation in subsequent iterations of image frames in the plurality of image frames; and
generating user interface data for display using the segmentation.

Assignees

• Snap Inc

Inventors

• Cowburn Piers
• Li David
• Müller Sandvik Isac Andreas
• Pan Qi
• Spek Andrew Tristan

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/174

  involving the use of two or more images · CPC title