View synthesis system and method using depth map

Patent metadata

Field	Value
Publication number	US-12567201-B2
Application number	US-202418412520-A
Country	US
Kind code	B2
Filing date	Jan 13, 2024
Priority date	Aug 3, 2021
Publication date	Mar 3, 2026
Grant date	Mar 3, 2026

Abstract

In multiview image generation and display, a computing device can synthesize view images of a multiview image of a scene from a color image and a depth map. Each view image can include color values at respective pixel locations. The computing device can render the view images of the multiview image on a multiview display. Synthesizing a view image can include, for a pixel location in the view image, the following operations. The computing device can cast a ray from the pixel location toward the scene in a direction corresponding to a view direction of the view image. The computing device can determine a ray intersection location at which the ray intersects a virtual surface specified by the depth map. The computing device can set a color value of the view image at the pixel location to correspond to a color of the color image at the ray intersection location.

First claim

What is claimed is:

1 . A computer-implemented method of multiview image generation and display, the method comprising:
receiving, with a computing device, a color image of a scene and a depth map of the scene; synthesizing, with the computing device, view images of a multiview image of the scene from the color image and the depth map, the view images representing the scene from different view directions, each view image including pixel locations and respective color values at the pixel locations; and rendering the view images of the multiview image on a multiview display of the computing device, wherein synthesizing a view image of the view images of the multiview image comprises, for a pixel location of the pixel locations in the view image:
casting a ray from the pixel location toward the scene in a direction corresponding to the view direction of the view image;
determining a ray intersection location at which the ray intersects a virtual surface specified by the depth map; and
setting a color value of the respective color values of the view image at the pixel location to correspond to a color of the color image at the ray intersection location; and
wherein determining the ray intersection location comprises:
determining sequential provisional locations along the ray between the pixel location and a specified plane such that the virtual surface lies between the pixel location and the specified plane;
identifying a provisional location of the sequential provisional locations along the ray comprising:
determining that the identified provisional location lies between the pixel location and the specified plane; and
advancing the identified provisional location to a next provisional location along the ray;

determining that the virtual surface lies between the pixel location and the identified provisional location; and
setting the ray intersection location to correspond to a location between the identified provisional location and an adjacent and previously identified provisional location, inclusive.

2 . The computer-implemented method of claim 1 , wherein determining the ray intersection location further comprises:
determining sequential second provisional locations along the ray between the identified provisional location and the adjacent and previously identified provisional location; identifying a second provisional location of the sequential second provisional locations along the ray comprising: determining that the identified second provisional location lies between the pixel location and the specified plane; and advancing the identified second provisional location to a next second provisional location along the ray; determining that the virtual surface lies between the pixel location and the identified second provisional location; and setting the ray intersection location to correspond to a location between the identified second provisional location and an adjacent and previously-identified second provisional location, inclusive.

3 . The computer-implemented method of claim 1 , wherein the provisional locations are equally spaced along the ray.

4 . The computer-implemented method of claim 3 , wherein:
the view image defines a horizontal direction being parallel to a top and bottom edges of the view image, a vertical direction being in a plane of the view image and orthogonal to the horizontal direction, and a depth that is orthogonal to the horizontal and vertical directions; and the provisional locations are spaced such that a horizontal component of a spacing between adjacent provisional locations corresponds to a specified value.

5 . The computer-implemented method of claim 4 , wherein the specified value corresponds to a horizontal spacing between adjacent pixels in the view image.

6 . The computer-implemented method of claim 1 , wherein synthesizing a view image of the view images of the multiview image comprises, for the pixel location in the view image:
casting the ray from the pixel location toward the scene in the direction corresponding to the view direction of the view image; determining that the ray does not intersect the virtual surface specified by the depth map; retrieving color information from at least one time-adjacent video frame of the view images of the multiview image; and setting the color value of the respective color values of the view image at the pixel location to correspond to the retrieved color information.

7 . The computer-implemented method of claim 6 , wherein determining that the ray does not intersect the virtual surface specified by the depth map comprises determining that the ray has propagated away from the pixel location by a distance that exceeds a threshold distance.

8 . The computer-implemented method of claim 6 , wherein the view images correspond to sequential images of a video signal; and the color information is retrieved from the pixel location of the at least one time-adjacent video frame of the video signal.

9 . The computer-implemented method of claim 1 , wherein the different view directions lie in a horizontal plane that is parallel to top and bottom edges of the view images.

10 . A system configured to perform multiview image generation and display, the system comprising:
a multiview display; a central processing unit; and a memory that stores a plurality of instructions, which, when executed, cause the central processing unit to perform operations, the operations comprising: receiving a color image of a scene and a depth map of the scene; synthesizing view images of a multiview image of the scene from the color image and the depth map, the view images representing the scene from different view directions, each view image including pixel locations and respective color values at the pixel locations; and rendering the view images of the multiview image on the multiview display, wherein synthesizing a view image of the view images of the multiview image comprises, for a pixel location of the pixel locations in the view image:
casting a ray from the pixel location toward the scene in a direction corresponding to the view direction of the view image;
determining a ray intersection location at which the ray intersects a virtual surface specified by the depth map; and
setting a color value of the respective color values of the view image at the pixel location to correspond to a color of the color image at the ray intersection location; and
wherein determining the ray intersection location comprises:
determining sequential provisional locations along the ray between the pixel location and a specified plane such that the virtual surface lies between the pixel location and the specified plane;
identifying a provisional location of the sequential provisional locations along the ray comprising:
determining that the identified provisional location lies between the pixel location and the specified plane; and
advancing the identified provisional location to a next provisional location along the ray;

determining that the virtual surface lies between the pixel location and the identified provisional location; and
setting the ray intersection location to correspond to a location between the identified provisional location and an adjacent and previously identified provisional location, inclusive.

11 . The system of claim 10 , wherein determining the ray intersection location further comprises:
determining sequential second provisional locations along the ray between the identified provisional location and the adjacent and previously identified provisional location; identifying a second provisional location of the sequential second provisional locations along the ray compr

Assignees

• Leia Inc

Inventors

• Suprun Vadym

Classifications

• G06T2210/21

  Collision detection, intersection · CPC title

• G06T2207/20212

  Image combination · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title