Compression of multi-dimensional object representations

Patent metadata

Field	Value
Publication number	US-2020202575-A1
Application number	US-202016808247-A
Country	US
Kind code	A1
Filing date	Mar 3, 2020
Priority date	Sep 29, 2017
Publication date	Jun 25, 2020
Grant date	—

Abstract

Objects can be rendered in three dimensions and viewed and manipulated in an augmented reality environment. A number of object images, a number of segmentation masks, and an object mesh structure are used by a client device to render the object in three dimensions. The object images and segmentation masks can be sequenced into frames. The object images and segmentation masks can be partitioned into patches and sequenced, or ordered, within each patch, and a keyframe can be assigned in each patch. Then, the object images and segmentation masks can be encoded into video files and sent to a client device. The client device can quickly retrieve a requested object image and segmentation mask based at least in part on identifying the keyframe in the same patch as the object image and segmentation mask.

First claim

What is claimed is:

1 . A computer-implemented method, comprising:
under control of one or more computer systems configured with executable instructions, determining a first dimension and a second dimension of an object represented in a plurality of object images; determining, for the plurality of object images, a lowest first dimension and a lowest second dimension containing the object; cropping the plurality of object images into a plurality of cropped object images having the lowest first dimension and the lowest second dimension; ordering the plurality of object images into a respective plurality of frames within individual groups of a plurality of groups, such that every object image of the plurality of object images has a frame number; determining a keyframe for every group of the plurality of groups, the keyframe for a specified group being a reference frame for the specified group; encoding, in order of the frame number, the plurality of object images into an object image video file; receiving, from a client device, a request to display a representation of the object; creating a client device package including the object image video file; and sending the client device package for display on the client device.

2 . The computer-implemented method of claim 1 , wherein displaying the representation of the object on the client device comprises:
receiving a viewpoint request to view a representation of the object from a specified viewpoint; determining a corresponding frame associated with the specified viewpoint; and retrieving, based at least in part on the corresponding frame, the object image of the plurality of object images that is associated with the specified viewpoint.

3 . The computer-implemented method of claim 1 , the method further comprising:
creating a plurality of segmentation masks corresponding to the plurality of object images, individual segmentation masks of the plurality of segmentation masks including filled area associated with the object and unfilled area not associated with the object; and creating an object mesh structure based at least in part on the plurality of segmentation masks by subtracting the unfilled area from the individual segmentation masks of the plurality of segmentation masks from a three-dimensional object environment.

4 . The computer-implemented method of claim 1 , wherein the plurality of object images are captured by one or more cameras, and wherein the client device package further includes pose information for the plurality of object images, the pose information comprising:
a camera location component, the camera location component including a three-dimensional description of a location associated with a capturing camera for individual object images; and a camera orientation component including a three-dimensional description of an orientation for individual cameras of the one or more cameras that captured individual object images of the plurality of object images.

5 . The computer-implemented method of claim 1 , wherein every group of the plurality of groups includes a fixed number of frames, the fixed number being between 1 and 16 frames.

6 . A computer-implemented method, comprising:
under control of one or more computer systems configured with executable instructions, determining a first dimension and a second dimension of an object represented in a plurality of object images; determining, for the plurality of object images, a lowest first dimension and a lowest second dimension containing the object; cropping the plurality of object images into a plurality of cropped object images having the lowest first dimension and the lowest second dimension; modifying the plurality of cropped object images into a plurality of modified object images in which pixels not associated with the object are converted to a specified color value; and ordering the plurality of modified object images respectively into a plurality of frames, such that every modified object image of the plurality of modified object images has a frame number, the plurality of modified object images ordered by the frame number.

7 . The computer-implemented method of claim 6 , further comprising:
determining a plurality of keyframes from the plurality of frames, individual keyframes of the plurality of keyframes occurring at a fixed keyframe interval; and encoding the plurality of modified object images into a video file in order of the frame number.

8 . The computer-implemented method of claim 7 , further comprising:
creating a plurality of segmentation masks based at least in part on the plurality of object images, such that individual segmentation masks of the plurality of segmentation masks are respectively associated with individual object images of the plurality of object images, individual segmentation masks of the plurality of segmentation masks including filled area and unfilled area, the filled area comprising pixels associated with the object, the unfilled area comprising the pixels not associated with the object; ordering the plurality of segmentation masks into the respective plurality of frames, such that individual segmentation masks of the plurality of segmentation masks have an identical frame number as the individual object images of the plurality of object images that are respectively associated with the individual segmentation masks of the plurality of segmentation masks; and creating an object mesh structure based at least in part on the plurality of segmentation masks by subtracting the unfilled area from the individual segmentation masks of the plurality of segmentation masks from a three-dimensional object environment.

9 . The computer-implemented method of claim 8 , further comprising:
encoding the plurality of segmentation masks into a segmentation mask video file; creating a client device package including an object image video file, the segmentation mask video file, and the object mesh structure; receiving a request from a client device to view a representation of the object; sending the client device package to the client device; and displaying the representation of the object on the client device, the representation of the object including the object mesh structure overlaid by an object image of the plurality of object images that is associated with a first viewpoint.

10 . The computer-implemented method of claim 9 , wherein the plurality of object images are captured by one or more image-capture devices, and wherein the client device package further includes pose information for the plurality of object images, the pose information comprising:
an image-capture device location component, the image-capture device location component including a three-dimensional description of a location associated with a capturing image-capture device for individual object images; and an image-capture device orientation component including a three-dimensional description of an orientation for individual image-capture device of the one or more image-capture devices that captured individual object images of the plurality of object images.

11 . The computer-implemented method of claim 7 , wherein the fixed keyframe interval is between 1 and 16.

12 . A system for compressing images, comprising:
at least one processor; memory including instructions that, when executed by the at least one processor, cause the system to:
crop a plurality of object images into a plurality of cropped object images having a lowest first dimension and a lowest second dimension that can form a polygon that contains the object in all of the plurality of cropped object images;
order the plurality of object images

Assignees

• A9 Com Inc

Inventors

• Dhua Arnab Sanat Kumar
• Kumar Neil Raj
• Hillesland Karl
• Grzeszczuk Radek
• Robertson Scott Paul

Classifications

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T15/205 Primary

  Image-based rendering · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06T7/60

  Analysis of geometric attributes · CPC title

• G06T2207/20132

  Image cropping · CPC title