Systems and methods of multiview style transfer

Patent metadata

Field	Value
Publication number	US-12159372-B2
Application number	US-202217878010-A
Country	US
Kind code	B2
Filing date	Jul 31, 2022
Priority date	Mar 1, 2020
Publication date	Dec 3, 2024
Grant date	Dec 3, 2024

Abstract

A system and method of multiview style transfer apply a style transfer to individual views of a multiview image in a way that produces consistent results across all images. In some embodiments, the multiview style transfer includes receiving first and second images representative of first and second perspectives of a scene and first and second disparity maps corresponding to the first and second images, generating a first stylized image, generating a stylized shifted image based on the first stylized image and the first disparity map, generating a second stylized image based on a guided filter of the stylized shifted image and the second image, and generating a first and second stylized image based on the stylized shifted images and the disparity maps.

First claim

What is claimed is:

1. A multiview style transfer system comprising:
processing circuitry; and
a memory that includes instructions, the instructions, when executed by the processing circuitry, cause the processing circuitry to:
receive a first image representative of a first perspective of a scene, a first disparity map corresponding to the first image, and a second image representative of a second perspective of the scene;
generate a first stylized image representative of a style transfer model applied to the first image;
generate a stylized shifted image based on the first stylized image and the first disparity map, the stylized shifted image including a first shift of the first stylized image to the second perspective; and
generate a second stylized image based on a guided filter of the stylized shifted image and the second image, the guided filter to process edge characteristics in the second stylized image based on the second image.

2. The multiview style transfer system of claim 1 , further including a multiview display, wherein the instructions further cause the processing circuitry to:
generate the first stylized image based on the first disparity map;
generate the second stylized image based on a second disparity map corresponding to the second image, wherein the first stylized image and the second stylized image are configured to be concurrently rendered by a multiview display; and
display the first stylized image and the second stylized image on the multiview display.

3. The multiview style transfer system of claim 2 , wherein the instructions further cause the processing circuitry to:
synthesize a plurality of stylized perspective views based on the first stylized image and the second stylized image; and
display the plurality of stylized perspective views on the multi view display.

4. The multiview style transfer system of claim 2 , wherein the instructions further cause the processing circuitry to:
generate a first extrapolated image based on the first stylized image and the first disparity map, the first extrapolated image representing a first synthesized viewpoint extrapolated to a third viewpoint based on the first perspective of the scene associated with the first image;
generate a second extrapolated image based on the second stylized image and the second disparity map, the second extrapolated image representing a second synthesized viewpoint extrapolated to a fourth viewpoint based on the second perspective of the scene associated with the second image; and
display the first extrapolated image and the second extrapolated image on the multiview display.

5. The multiview style transfer system of claim 2 , wherein the instructions further cause the processing circuitry to:
generate a first stylized viewpoint image based on re-projecting the first stylized image from a first desired output viewpoint;
generate a second stylized viewpoint image based on re-projecting the second stylized image from a second desired output viewpoint; and
display the first stylized viewpoint image and the second stylized viewpoint image on the multiview display.

6. The multiview style transfer system of claim 5 , wherein:
the first desired output viewpoint is based on a first device viewpoint associated with a device multiview display; and
the second desired output viewpoint is based on a second device viewpoint associated with the device multiview display.

7. The multiview style transfer system of claim 5 , wherein the instructions further cause the processing circuitry to:
generate a third stylized viewpoint image based on re-projecting a third stylized image from a third desired output viewpoint;
generate a fourth stylized viewpoint image based on re-projecting a fourth stylized image from a fourth desired output viewpoint; and
display the third stylized viewpoint image and the fourth stylized viewpoint image on the multiview display.

8. The multiview style transfer system of claim 1 , wherein the instructions further cause the processing circuitry to generate the second stylized image as a second filtered image based on application of a second guided filter to the second stylized image, wherein the generation of the second filtered image is further based on the second image.

9. The multiview style transfer system of claim 8 , wherein the instructions further cause the processing circuitry to generate the first stylized image as a first filtered image based on application of a first guided filter to the first stylized image, wherein the generation of the first filtered image is further based on the first image.

10. The multiview style transfer system of claim 9 , wherein:
the first guided filter includes a first guided sharpening filter, the first guided sharpening filter to sharpen the first stylized image based on a first plurality of edges within the first image; and
the second guided filter includes a second guided sharpening filter, the second guided sharpening filter to sharpen the second stylized image based on a second plurality of edges within the second image.

11. The multiview style transfer system of claim 1 , wherein the instructions further cause the processing circuitry to generate the first stylized image based on the first image and the style transfer model.

12. The multiview style transfer system of claim 11 , wherein the style transfer model includes a style transfer neural network trained via machine learning on a plurality of target style images.

13. The multiview style transfer system of claim 11 , wherein the instructions further cause the processing circuitry to:
receive a target style image;
identify a target style based on application of a neural style transfer (NST) algorithm to the target style image; and
generate the style transfer model based on the target style.

14. The multiview style transfer system of claim 1 , wherein the first disparity map is generated based on the first image and the second image, the first disparity map representing differences in horizontal coordinates of a first plurality of image points in the first image relative to a second plurality of image points in the second image; and
wherein a second disparity map is generated based on the first image and the second image, the second disparity map representing differences in horizontal coordinates of the second plurality of image points in the second image relative to the first plurality of image points in the first image.

15. A multiview style transfer method comprising:
receiving a first image representative of a first perspective of a scene, a first disparity map corresponding to the first image, and a second image representative of a second perspective of the scene;
generating a first stylized image representative of a style transfer model applied to the first image;
generating a stylized shifted image based on the first stylized image and the first disparity map, the stylized shifted image including a first shift of the first stylized image to the second perspective; and
generating a second stylized image based on a guided filter of the stylized shifted image and the second image, the guided filter to process edge characteristics in the second stylized image based on the second image.

16. The multiview style transfer method of claim 15 , further comprising:
generating a first stylized image based on the first disparity map; and
generating a second stylized image based on a second disparity map corresponding to the second image, wherein the first stylized image and the second stylized image are configured to be rendered by a multiview display.

Assignees

• Leia Inc

Inventors

• Dahlquist Nicolas
• Gunaseelan Saravana
• Kohli Puneet
• Li Edward

Classifications

• G06N3/08

  Learning methods · CPC title

• G06T5/73

  Deblurring; Sharpening · CPC title

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title

• G06T2207/20192

  Edge enhancement; Edge preservation · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title