Methods and apparatus for mapping at least one received image to a surface of a model in a manner that efficiently uses the image content as a texture

Patent metadata

Field	Value
Publication number	US-10075701-B2
Application number	US-201615046331-A
Country	US
Kind code	B2
Filing date	Feb 17, 2016
Priority date	Feb 17, 2015
Publication date	Sep 11, 2018
Grant date	Sep 11, 2018

Abstract

Methods and apparatus for using selective resolution reduction on images to be transmitted and/or used by a playback device are described. Prior to transmission one or more images of an environment are captured. Based on image content, motion detection and/or user input a resolution reduction operation is selected and performed. The reduced resolution image is communicated to a playback device along with information indicating a UV map corresponding to the selected resolution allocation that should be used by the playback device for rendering the communicated image. By changing the resolution allocation used and which UV map is used by the playback device different resolution allocations can be made with respect to different portions of the environment while allowing the number of pixels in transmitted images to remain constant. The playback device renders the individual images with the UV map corresponding to the resolution allocation used to generate the individual images.

First claim

What is claimed:

1. A content playback method comprising:
receiving a first encoded image, said first encoded image being an image which was generated by performing a first non-uniform downsampling operation on an image and encoding the downsampled image, said first non-uniform downsampling operation being a downsampling operation in which at least one image portion is downsampled more than another image portion;
decoding the first encoded image to generate a first decoded image;
mapping the first decoded image to a mesh model of an environment in accordance with a first image map corresponding to said first non-uniform downsampling operation to produce a first rendered image, said first image map mapping different numbers of pixels of the decoded image to different segments of said mesh model of the environment; and
displaying the first rendered image on a display device;
receiving a signal indicating that a second image map, corresponding to a second non-uniform downsampling operation, should be used to map portions of additional received images to said mesh model of the environment, said second non-uniform downsampling operation being different from said first non-uniform downsampling operation;
wherein said first image map allocates a first number of pixels of the first decoded image to a first segment of said mesh model of the environment; and
wherein said second image map allocates a second number of pixels of an additional image to the first segment of said mesh model of the environment as part of an additional image rendering operation used to generate an additional rendered image, the first and second number of pixels being different.

2. The method of claim 1 , wherein the different numbers of pixels are mapped to environmental regions of the same size but located at different locations in the environment.

3. The method of claim 2 , wherein segments in said environment corresponding to action are allocated more pixels than segments in which less or no action is detected.

4. The method of claim 1 , wherein at least some segments corresponding to a front viewing area are allocated more pixels per segment than segments corresponding to a rear viewing area.

5. The method of claim 1 ,
wherein said first decoded image is a first frame.

6. The method of claim 5 , wherein said first image map maps a first size image portion of the first decoded image to said first segment of said mesh model of the environment and the second image map maps a second size image segment to said first segment of said mesh model of the environment, said first and second size image segments including different numbers of pixels.

7. The method of claim 1 , wherein said first encoded image is an image of an environment that was captured by a camera.

8. The method of claim 1 , wherein said first encoded image is one of a left or right eye image captured by a corresponding camera of a stereoscopic camera pair including a left camera and a right camera.

9. A content playback apparatus comprising:
a receiver for receiving a first encoded image, said first encoded image being an image which was generated by performing a first non-uniform downsampling operation on an image and encoding the downsampled image, said first non-uniform downsampling operation being a downsampling operation in which at least one image portion is downsampled more than another image portion, said receiver also being for receiving a signal indicating that a second image map, corresponding to a second non-uniform downsampling operation, should be used to map portions of additional received images to an environmental mesh model, said second non-uniform downsampling operation being different from said first non-uniform downsampling operation;
wherein said first image map allocates a first number of pixels of the first decoded image to a first segment of said environmental mesh model; and
wherein said second image map allocates a second number of pixels of an additional image to the first segment of said environmental mesh model as part of an additional image rendering operation used to generate an additional rendered image, the first and second number of pixels being different;
a decoder for decoding the first encoded image to generate a first decoded image;
a processor configured to map the first decoded image to said environmental mesh model in accordance with the first image map to produce a first rendered image, said first image map mapping different numbers of pixels of the decoded image to different segments of said environmental mesh model; and
a display for displaying rendered images.

10. The apparatus of claim 9 , wherein the different numbers of pixels are mapped to environmental regions of the same size but located at different locations in the environment.

11. The apparatus of claim 9 , wherein segments in said environment corresponding to action are allocated more pixels than segments in which less or no action is detected.

12. The apparatus of claim 9 ,
wherein said first decoded image is a frame.

13. A non-transitory computer readable medium comprising processor executable instructions, which when executed by a processor, control a content playback device to:
receive a first encoded image, said first encoded image being an image which was generated by performing a first non-uniform downsampling operation on an image and encoding the downsampled image, said first non-uniform downsampling operation being a downsampling operation in which at least one image portion is downsampled more than another image portion;
decode the first encoded image to generate a first decoded image;
map the first decoded image to a mesh model of an environment in accordance with a first image map corresponding to said first non-uniform downsampling operation to produce a first rendered image, said first image map mapping different numbers of pixels of the decoded image to different segments of said mesh model of the environment;
display the first rendered image on a display device; and
receive a signal indicating that a second image map, corresponding to a second non-uniform downsampling operation, should be used to map portions of additional received images to said mesh model of the environment, said second non-uniform downsampling operation being different from said first non-uniform downsampling operation;
wherein said first image map allocates a first number of pixels of the first decoded image to a first segment of said mesh model of the environment; and
wherein said second image map allocates a second number of pixels of an additional image to the first segment of said mesh model of the environment as part of an additional image rendering operation used to generate an additional rendered image, the first and second number of pixels being different.

Assignees

• Nextvr Inc

Inventors

• Cole David
• Moss Alan Mckay
• Medina Hector M

Classifications

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N19/597

  specially adapted for multi-view video sequence encoding · CPC title

• H04N21/8146

  involving graphical data, e.g. 3D object, 2D graphics · CPC title

• H04N19/172

  the region being a picture, frame or field · CPC title

• H04N13/194 Primary

  Transmission of image signals · CPC title