Automatic 3d camera alignment and object arrangment to match a 2d background image

Patent metadata

Field	Value
Publication number	US-2019139319-A1
Application number	US-201715804908-A
Country	US
Kind code	A1
Filing date	Nov 6, 2017
Priority date	Nov 6, 2017
Publication date	May 9, 2019
Grant date	—

Abstract

Embodiments disclosed herein provide systems, methods, and computer storage media for automatically aligning a 3D camera with a 2D background image. An automated image analysis can be performed on the 2D background image, and a classifier can predict whether the automated image analysis is accurate within a selected confidence level. As such, a feature can be enabled that allows a user to automatically align the 3D camera with the 2D background image. For example, where the automated analysis detects a horizon and one or more vanishing points from the background image, the 3D camera can be automatically transformed to align with the detected horizon and to point at a detected horizon-located vanishing point. In some embodiments, 3D objects in a 3D scene can be pivoted and the 3D camera dollied forward or backwards to reduce changes to the framing of the 3D composition resulting from the 3D camera transformation.

First claim

What is claimed is:

1 . A computerized method for automatically aligning a 3D virtual camera of a 3D scene with a 2D image, the method comprising:
automatically aligning the 3D camera with the 2D image by applying a 3D transformation to the 3D camera to point the 3D camera at a detected horizon of the 2D image and change a perspective from which to view 3D objects in the 3D scene, wherein the perspective is associated with a 2D viewport frame corresponding to the 3D camera; and pivoting one or more 3D objects in the 3D scene around a pivot point by a pivoting angle determined by a rotation of the 3D camera corresponding to the 3D transformation.

2 . The method of claim 1 , wherein the automatically aligning the 3D camera with the 2D image comprises:
superimposing the 2D image on the 2D viewport frame; generating a viewport horizon in the 2D viewport frame corresponding to the detected horizon; generating a target orientation matrix for the 3D camera corresponding to a target orientation pointing the 3D camera at the viewport horizon; and applying the 3D transformation to the 3D camera based on the target orientation matrix.

3 . The method of claim 1 , additionally comprising:
using a classifier to predict an accuracy of at least one of the detected horizon or a detected vanishing point for the 2D image, and wherein the 3D transformation is only applied if the classifier predicts the accuracy.

4 . The method of claim 3 :
wherein the detected vanishing point is a detected horizon-located vanishing point corresponding to the detected horizon; and wherein the classifier is configured to predict the accuracy of the detected horizon based on detected image features from the 2D image, the detected image features comprising statistics about detected image line segments relative to the detected horizon-located vanishing point.

5 . The method of claim 1 , additionally comprising:
generating a target field of view for the 3D camera to match a visible portion of the 2D image in the 2D viewport frame; applying the target field of view to the 3D camera; and translating the 3D camera to compensate for the applied target field of view.

6 . The method of claim 1 ,
wherein the 3D transformation is based on a target yaw of the 3D camera; wherein the method additionally comprises generating a delta yaw that would result from rotating the 3D camera to the target yaw; and wherein the pivoting angle is the delta yaw.

7 . The method of claim 2 , wherein the generating the target orientation matrix comprises:
generating an intermediate orientation matrix that would point the 3D camera at the viewport horizon; generating a target yaw that would point the 3D camera at a detected vanishing point of the 2D image in the 2D viewport frame; and determining the target orientation matrix based on the intermediate orientation matrix and the target yaw.

8 . The method of claim 7 , wherein the intermediate orientation matrix comprises:
a forward vector corresponding to a ray cast from the 3D camera through a center point of the viewport horizon; an up vector corresponding to the forward vector crossed with a vector corresponding to a ray cast from the 3D camera through an edge point of the viewport horizon; and a right vector corresponding to the up vector crossed with the forward vector.

9 . The method of claim 8 , wherein the generating the target yaw comprises:
generating a forward vanishing point ray by casting a ray from the 3D camera to the detected vanishing point; projecting the forward vanishing point ray onto a 3D ground plane; and determining the target yaw based on the angle between the forward vector and the projected forward vanishing point ray.

10 . The method of claim 1 , additionally comprising:
generating a pivot plane that intersects a center of a 3D scene bounding box normal to an original forward vector of the 3D camera; and generating the pivot point as a point of intersection between the pivot plane and the original forward vector.

11 . One or more computer storage media storing computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations for automatically aligning a virtual 3D camera of a 3D scene with a 2D image, the operations comprising:
superimposing the 2D image on a 2D viewport frame corresponding to the 3D camera; generating a viewport horizon in the 2D viewport frame corresponding to a detected horizon of the 2D image; generating a target orientation matrix for the 3D camera corresponding to a target orientation pointing the 3D camera at the viewport horizon; applying a 3D transformation to the 3D camera based on the target orientation matrix; and pivoting one or more 3D objects in the 3D scene around a pivot point by a pivoting angle determined by a rotation of the 3D camera corresponding to the 3D transformation.

12 . The media of claim 11 ,
wherein the operations additionally comprise using a classifier to predict an accuracy of at least one of the detected horizon or a detected vanishing point for the 2D image, and wherein the 3D transformation is only applied if the classifier predicts the accuracy.

13 . The media of claim 12 ,
wherein the detected vanishing point is a detected horizon-located vanishing point corresponding to the detected horizon; and wherein the classifier is configured to predict the accuracy of the detected horizon based on detected image features from the 2D image, the detected image features comprising statistics about detected image line segments relative to the detected horizon-located vanishing point.

14 . The media of claim 11 , wherein the operations additionally comprise:
generating a target field of view for the 3D camera to match a visible portion of the 2D image in the 2D viewport frame; applying the target field of view to the 3D camera; and translating the 3D camera to compensate for the applied target field of view.

15 . The media of claim 11 ,
wherein the target orientation matrix is based on a target yaw of the 3D camera; wherein the operations additionally comprise generating a delta yaw that would result from rotating the 3D camera to the target yaw; and wherein the pivoting angle is the delta yaw.

16 . A computer system comprising:
one or more hardware processors and memory configured to provide computer program instructions to the one or more hardware processors; means for determining a prediction that an automated image analysis comprising at least one of a detected horizon or a detected vanishing point for a 2D image is likely to be accurate; an alignment tool configured to utilize the one or more hardware processors to perform operations when the means for determining a prediction predicts that the automated image analysis is likely to be accurate, the operations comprising:
automatically aligning a virtual 3D camera with the 2D image by applying a 3D transformation to the 3D camera to point the 3D camera at the detected horizon of the 2D image and change a perspective from which to view 3D objects in a 3D scene, wherein the perspective is associated with a 2D viewport frame corresponding to the 3D camera; and
pivoting one or more 3D objects in the 3D scene around a pivot point by a pivoting angle determined by a rotation of the 3D camera corresponding to the 3D transformation.

17 . The computer system of claim 16 , wherein the alignment tool is configured to automatically align the 3D camera with the 2D image by:
superimposing the 2D image o

Assignees

• Adobe Systems Inc

Inventors

• Eisenmann Jonathan
• Oxholm Geoffrey Alan
• Shechtman Elya
• Russell Bryan

Classifications

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T2219/2016

  Rotation, translation, scaling · CPC title

• G06T2219/2004

  Aligning objects, relative positioning of parts · CPC title

• G06T15/20

  Perspective computation · CPC title