Stylized image painting

Patent metadata

Field	Value
Publication number	US-11030793-B2
Application number	US-201916587015-A
Country	US
Kind code	B2
Filing date	Sep 29, 2019
Priority date	Sep 29, 2019
Publication date	Jun 8, 2021
Grant date	Jun 8, 2021

Abstract

A photo filter (e.g., artistic/stylized painting) light field effect system includes an eyewear device having a frame, a temple connected to a lateral side of the frame, and a depth-capturing camera. Execution of programming by a processor configures the stylized image painting effect system to apply a photo filter selection to: (i) a left raw image or a left processed image to create a left photo filter image, and (ii) a right raw image or a right processed image to create a right photo filter image. The stylized image painting effect system generates a photo filter stylized painting effect image with an appearance of a spatial rotation or movement, by blending together the left photo filter image and the right photo filter image based on a left image disparity map and a right image disparity map.

First claim

What is claimed is:

1. A stylized painting effect system for creating a stylized painting effect image, the system comprising:
an eyewear device including:
a frame having a temple connected to a lateral side of the frame; and
a depth-capturing camera configured to capture at least one of a left raw image or a right raw image;

an image display for presenting images, including an original image, wherein the original image is based on the left raw image, a left processed image, the right raw image, a right processed image, or combination thereof;
an image display driver coupled to the image display to control the image display to present the original image;
a user input device to receive mark-ups for the original image, a stylized painting effect selection, and a style selection from a user;
a memory and a processor coupled to the depth-capturing camera, the image display driver, the user input device, and the memory; and
programming in the memory, wherein execution of the programming by the processor configures the stylized painting effect system to perform functions, including functions to:
capture, via the depth-capturing camera, at least one of the left raw image or the right raw image;
present, via the image display, the original image;
receive, via the user input device, the markups, the stylized painting effect selection, and the style selection from the user;
create at least one stylized painting effect image with a stylized painting effect scene;
apply the stylized painting effect image to the mark-ups in: (i) the left raw image or the left processed image to create a left stylized painting effect image, (ii) the right raw image or the right processed image to create a right stylized painting effect image, or (iii) combination thereof;
generate a stylized painting effect image having an appearance of a spatial movement or rotation around the stylized painting effect scene of the at least one stylized painting effect image, by blending together the left stylized painting effect image and the right stylized painting effect image; and
present, via the image display, the stylized painting effect image.

2. The system of claim 1 , wherein the function to receive the style selection comprises functions to:
receive a selection of an image from a user; and
wherein the function to create the at least one stylized painting effect image with the stylized painting effect scene comprises a function to:
apply the image selection to the markups using neural style transfer.

3. The system of claim 1 , wherein the function to create the at least one stylized painting effect image comprises the function to:
blend the stylized painting effect image with the original image in the regions of the mark-ups.

4. The system of claim 1 , wherein the user input device comprises:
an inertial measurement unit configured to develop the spatial movement or rotation around the stylized painting effect scene of the at least one stylized painting effect image.

5. The system of claim 1 , further including a function to:
calculate: (i) a left image disparity map between a left pixel matrix of pixels and a right pixel matrix of pixels, and (ii) a right image disparity map between the right pixel matrix and the left pixel matrix for use in blending together the left stylized painting effect image and the right stylized painting effect image.

6. The stylized painting effect system of claim 5 , wherein the function to generate the stylized painting effect image includes:
determining a horizontal position movement parameter along an X axis of the left pixel matrix and the right pixel matrix;
filling up a left interpolated pixel matrix by moving pixels in the left pixel matrix along the X axis based on the horizontal movement parameter;
filling up a right interpolated pixel matrix by moving pixels in the right pixel matrix along the X axis based on the horizontal movement parameter; and
creating the stylized painting effect image by blending together the left interpolated pixel matrix and the right interpolated pixel matrix.

7. The stylized painting effect system of claim 6 , wherein:
the function of generating the stylized painting effect image by blending together the left interpolated pixel matrix and the right interpolated pixel matrix is based on disparity confidence levels, gradients, or combination thereof in the left image disparity map and the right image disparity map.

8. The stylized painting effect system of claim 6 , wherein:
determining the horizontal position movement parameter includes:
receiving, via the user input device, a two-dimensional input selection of the presented original image from the user; and
tracking, via the user input device, motion of the two-dimensional input selection from an initial touch point to a final touch point of the presented original image.

9. The stylized painting effect system of claim 8 , wherein:
the user input device includes:
a touch sensor including an input surface and a sensor array that is coupled to the input surface to receive at least one finger contact inputted from a user; and
a sensing circuit integrated into or connected to the touch sensor and connected to the processor, the sensing circuit configured to measure voltage to track the at least one finger contact on the input surface;

the function of receiving, via the user input device, the stylized painting effect selection from the user includes receiving, on the input surface of the touch sensor, the at least one finger contact inputted from the user; and
the function of tracking, via the user input device, motion of the two-dimensional input selection from the initial touch point to the final touch point to derive the horizontal position movement parameter, includes tracking, via the sensing circuit, drag from the at least one finger contact on the input surface from the initial touch point to the final touch point on the input surface of the touch sensor.

10. A method comprising steps of:
capturing, via a depth-capturing camera, at least one of a left raw image and a right raw image;
presenting, via an image display, an original image, wherein the original image is based on the left raw image, the left processed image, the right raw image, the right processed image, or combination thereof;
receiving, via a user input device, markups, a stylized painting effect selection, and a style selection from the user;
creating at least one stylized painting effect image with a stylized painting effect scene;
applying the at least one stylized painting effect image to the mark-ups in: (i) the left raw image or the left processed image to create a left stylized painting effect image, (ii) the right raw image or the right processed image to create a right stylized painting effect image, or (iii) combination thereof;
generating a stylized painting effect image having an appearance of a spatial movement or rotation around the stylized painting effect scene of the at least one stylized painting effect image, by blending together the left stylized painting effect image and the right stylized painting effect image; and
presenting, via the image display, the stylized painting effect image.

11. The method of claim 10 , wherein the receiving comprises:
receiving a selection of an image from a user; and
wherein the creating comprises:
applying the image selection to the markups using a neural style transfer.

12. The method of claim 10 , further comprising:
determining inertial movement of the image display to develop the spatial movement or rotation around the stylized painting effect scene of the at least one stylized painting effect image.

Assignees

• Katz Sagi
• Snap Inc

Inventors

• Katz Sagi

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title

• H04N13/344

  with head-mounted left-right displays · CPC title

• H04N13/271

  wherein the generated image signals comprise depth maps or disparity maps · CPC title

• H04N13/254

  in combination with electromagnetic radiation sources for illuminating objects · CPC title