Centralized database for 3-D and other information in videos

Patent metadata

Field	Value
Publication number	US-9542975-B2
Application number	US-91168810-A
Country	US
Kind code	B2
Filing date	Oct 25, 2010
Priority date	Oct 25, 2010
Publication date	Jan 10, 2017
Grant date	Jan 10, 2017

Abstract

Methods and systems for a centralized database for 3-D and other information in videos are presented. A centralized database contains video metadata such as camera, lighting, sound, object, depth, and annotation data that may be queried for and used in the editing of videos, including the addition and removal of objects and sounds. The metadata stored in the centralized database may be open to the public and admit contributor metadata.

First claim

What is claimed is:

1. A method comprising:
accessing a first version of video data captured by a camera to use in post capture processing by a video player, the video data comprising a video identifier;
querying a server, with the video identifier by the video player, for video metadata stored within a centralized database on the server, the video metadata comprising video lighting location metadata with coordinates of a light source with respect to the camera that captured the video data, and wherein the video metadata comprises time data indicating time information corresponding to the coordinates of the light source during the capture of the video data;
receiving the video metadata by the video player from the server;
editing the first version of the video data with the received video metadata by the video player to produce a second version of the video data, wherein editing the first version of the video data to produce the second version of the video data includes:
determining, based on the time data and the coordinates of the light source, a change in the light source during the capture of the video data; and
producing the second version of the video data based on the first version of the video data, the second version of the video data produced by adjusting light provided by the light source for the second version of the video data for a time period during the capture of the video data, the light being adjusted using the determined change in the light source, wherein adjusting the light provided by the light source for the second version of the video data includes:
adjusting the light for each object of a plurality of objects in the second version of the video data, wherein the light for the object is adjusted based on object segmentation data and depth map data included in the video metadata for the object, and wherein the light for the object is adjusted by:
creating a set of reference images of the object;
rotating the light source for the light provided for each reference image in the set of reference images;
generating a composite image of the object based on the set of reference images after rotating the light source, wherein the composite image includes information about a location of the light source for the reference image after rotating the light source and includes information about a reflectance of the object;
wherein the light for the object in the second version of the video data is adjusted based on the composite image generated for the object, and wherein the composite image of the object indicates an animated progression of a movement of the light source with respect to the object in the image; and

outputting the second version of the video data to a display by the video player while accessing the first version of the video data.

2. The method as in claim 1 wherein the video metadata comprises lighting data, sound data, camera data, the object segmentation data, the depth map data, composition data, and annotation data.

3. The method as in claim 1 wherein editing the first version of the video data includes:
analyzing the object segmentation data, the depth map data, and annotation data included in the video metadata to identify one or more objects to remove from the first version of the video data to produce the second version of the video data after the capture of the first version of the video data by the camera; and
removing the one or more objects from the first version of the video data to produce the second version of the video data after the capture of the first version of the video data by the camera; and
wherein the video lighting location metadata is applied to determine a background behind the removed one or more objects in the second version of the video data, the background being consistent with a background identified based on the video lighting location metadata throughout the capture of the first version of the video data.

4. The method as in claim 1 wherein editing the first version of the video data includes:
adding one or more objects to the first version of the video data to produce the second version of the video data, wherein the one or more objects is added after the capture of the first version of the video data by the camera; and
wherein the video lighting location metadata is applied to the added one or more objects to cause the added one or more objects to be lighted by the light source.

5. The method as in claim 1 wherein the change in the light source includes a change in lighting provided by the light source, and wherein the second version of the video data is produced upon assessing that the change in the lighting provided by the light source satisfies a lighting condition.

6. The method as in claim 1 wherein editing the first version of the video data includes:
analyzing camera data and location data to combine the first version of the video data captured by two or more cameras for a scene; and
creating a pair of stereoscopic images to present a three dimensional image of the scene in the second version of the video data.

7. The method as in claim 1 wherein editing the first version of the video data includes:
removing a sound from the first version of the video data to produce the second version of the video data; and
wherein the video metadata is applied so that the removed sound is consistent with the sound data.

8. The method as in claim 3 wherein editing the first version of the video data includes:
identifying, using the object segmentation data, the depth map data, and the annotation data, the one or more objects to be removed.

9. The method as in claim 1 wherein the video metadata is stored at a granularity of a single frame in the centralized database on the server.

10. The method as in claim 2 wherein the sound data comprises sound location metadata including coordinates of a sound source, and editing of the first version of the video data incorporates the sound location metadata into the second version of the video data.

11. The method as in claim 1 wherein the change in lighting provided by the light source is from a first time period to a second time period, the method further comprising:
assessing whether the change in the lighting provided by the light source from the first time period to the second time period satisfies the lighting condition, the lighting condition being a lighting quality condition;
wherein adjusting the light provided by the light source occurs for the first time period during the capture of the video data, and wherein the light is adjusted based on light provided by the light source for the second time period.

12. The method as in claim 1 , wherein the light for the object is adjusted by rotating the light source to a different position for each of a set of reference images that are stationary in the video data that is captured.

13. The method as in claim 1 , wherein the composite image is generated using a polynomial texture map, and wherein the composite image is a parametric photograph.

14. A system comprising:
a video player that performs operations comprising:
accessing a first version of video data, the first version of the video data comprising a video identifier;
querying a server with the video identifier for video metadata stored within a centralized database on the server, the video metadata comprising video lighting location metadata with coordinates of a light source with respect to a camera that captured the video data, and wherein the video metadata comprises time data indicating time information corresponding to the coordinates of the light source during capture of the video data;
receiv

Assignees

• Osman Steven
• Stamate Vlad
• Sony Interactive Entertainment Inc

Inventors

• Osman Steven
• Stamate Vlad

Classifications

• G11B27/034 Primary

  on discs (G11B27/036, G11B27/038 take precedence) · CPC title

• G11B27/036

  Insert-editing · CPC title