Visualizing video within existing still images

Patent metadata

Field	Value
Publication number	US-9594960-B2
Application number	US-88170410-A
Country	US
Kind code	B2
Filing date	Sep 14, 2010
Priority date	Sep 14, 2010
Publication date	Mar 14, 2017
Grant date	Mar 14, 2017

Abstract

Video from a video camera can be integrated into a still image, with which it shares common elements, to provide greater context and understandability. Pre-processing can derive transformation parameters for transforming and aligning the video to be integrated into the still image in a visually fluid manner. The transformation parameters can then be utilized to transform and align the video in real-time and display it within the still image. Pre-processing can comprise stabilization of video, if the video camera is moveable, and can comprise identification of areas of motion and of static elements. Transformation parameters can be derived by fitting the static elements of the video to portions of one or more existing images. Display of the video in real-time in the still image can include display of the entire transformed and aligned video image, or of only selected sections, to provide for a smoother visual integration.

First claim

We claim:

1. A computing device comprising:
one or more processing units; and
one or more computer-readable storage media comprising computer-executable instructions which, when executed by the one or more processing units, cause the computing device to:
obtain a sample of a video image feed;
select one or more pre-existing still images whose image scope comprises at least a portion of an image scope of the obtained sample of the video image feed, the selecting being informed by location metadata associated with both the video image feed and the still images;
derive an average image comprising those elements that remain static throughout the sample of the video image feed;
derive a motion mask identifying areas in which elements move throughout the sample of the video image feed;
identifying image features in the average image that are along the motion mask, having corresponding image features in the selected one or more still images; and
derive transformation parameters to transform and align the video image feed such that the identified image features in the sample of the video image feed, after transformation and alignment, have a visual size and visual appearance equivalent to that of the corresponding image features in the selected one or more still images, and can be equivalently overlaid over the corresponding image features in the selected one or more still images.

2. The computing device of claim 1 , wherein the video image feed is from a traffic camera and wherein further the pre-existing still images are panoramas providing a street-level view.

3. The computing device of claim 1 , wherein the computer-executable instructions for identifying the image features in the video image feed comprise computer-executable instructions for identifying image features in the selected one or more still images that are along a roadway, associated with the selected one or more still images, along which a traffic camera is oriented, wherein the video image feed is from the traffic camera, and wherein the still images are panoramas.

4. The computing device of claim 1 , wherein the computer-executable instructions for deriving the transformation parameters comprise computer-executable instructions for: selecting a first set of image features from among the identified image features in the sample of the video image feed; deriving preliminary transformation parameters; transforming and aligning the sample of the video image feed with the derived preliminary transformation parameters; scoring an overall fit between the transformed and aligned sample of the video image feed and the at least one of the selected still images, the scoring being based on an accuracy of alignment of others of the identified image features that are not part of the first set of image features; repeating the deriving and the scoring for subsequent sets of image features, selected from among the identified image features in the sample of the video image feed; and selecting, as the transformation parameters, those preliminary transformation parameters that yielded a best score of the overall fit.

5. The computing device of claim 1 , wherein the identified image features are traffic lane marking lines and wherein further the computer-executable instructions for deriving the transformation parameters comprise computer-executable instructions for performing homography utilizing the traffic lane marking lines.

6. The computing device of claim 1 , wherein the one or more computer-readable storage media comprise further computer-executable instructions, which, when executed by the one or more processing units, cause the computing device to: stabilize the sample of the video image feed into a composite with individual frames of the sample of the video image feed aligned with one another such that common image elements of each frame overlap.

7. The computing device of claim 1 , wherein the one or more computer-readable storage media comprise further computer-executable instructions, which, when executed by the one or more processing units, cause the computing device to: transform and align the video image feed while it is being received utilizing the transformation parameters; and generate a continuously updated amalgamated image comprising at least a portion of an image scope of the transformed and aligned video image feed overlaid over at least one of the selected still images such that the portion of the image scope of the transformed and aligned video image feed is equivalent to an image scope of that portion of the least one of the selected still images over which the transformed and aligned video image feed is overlaid.

8. A computing device comprising:
one or more processing units; and
one or more computer-readable storage media comprising computer-executable instructions which, when executed by the one or more processing units, cause the computing device to:
transform and align a video image feed, while it is being received, utilizing transformation parameters; and
generate a continuously updated amalgamated image comprising at least a portion of an image scope of the transformed and aligned video image feed overlaid over at least one pre-existing still image, which was selected based on location metadata associated with both the video image feed and the at least one pre-existing still image, such that the portion of the image scope of the transformed and aligned video image feed is equivalent to an image scope of that portion of the least one pre-existing still image over which the transformed and aligned video image feed is overlaid, wherein the transformed and aligned video image feed comprises image features having a visual size and appearance equivalent to that of corresponding images features in the least one pre-existing still image;
wherein the transformation parameters are derived based on image features in an average image that are along a motion mask, the image features having corresponding image features in the at least one pre-existing still image, wherein the average image comprises those elements that remain static throughout a sample of the video image feed, and wherein further the motion mask identifies areas in which elements move through the sample of the video image feed.

9. The computing device of claim 8 , wherein the video image feed is from a traffic camera and wherein further the at least one pre-existing still image is a panorama.

10. The computing device of claim 9 , wherein the computer-executable instructions for generating the amalgamated image comprise computer-executable instructions for: filtering out portions of the video image feed other than a roadway and vehicular traffic thereon and overlaying only the roadway and the vehicular traffic onto the at least one pre-existing still image.

11. The computing device of claim 8 , wherein the computer-executable instructions for generating the amalgamated image comprise computer-executable instructions for: filtering out static image elements from the video image feed and overlaying only moving image elements from the video image feed onto the at least one pre-existing still image.

12. The computing device of claim 8 , wherein the at least one pre-existing still image has associated with it location metadata identifying a location proximate to a location identified by location metadata associated with the video image feed.

13. A method for generating an amalgamated image comprising a video image feed visually integrated into a pre-existing still image, the method comprising the steps of:
obtaining a sample of the video image feed;
selecting the pre-existing still image based upon its image scope comprising at least a port

Assignees

• Ofek Eyal
• Chen Billy
• Microsoft Technology Licensing Llc

Inventors

• Ofek Eyal
• Chen Billy

Classifications

• H04N23/6845

  by combination of a plurality of images sequentially taken · CPC title

• H04N5/272

  Means for inserting a foreground image in a background image, i.e. inlay, outlay · CPC title

• G06T2207/30256

  Lane; Road marking · CPC title

• G06T7/33

  using feature-based methods · CPC title

• G08G1/04 Primary

  using optical or ultrasonic detectors · CPC title